調用Pretrained TOK跟NER的時候報錯
chanandrew96 opened this issue · comments
Describe the bug
調用部分Pretrained TOK跟NER的時候報錯，已試過直接調用已下載到本地的Model也是出現相同的報錯
NER:
hanlp.pretrained.ner.CONLL03_NER_BERT_BASE_CASED_EN
hanlp.pretrained.ner.MSRA_NER_ALBERT_BASE_ZH
hanlp.pretrained.ner.MSRA_NER_BERT_BASE_ZH
TOK:
hanlp.pretrained.tok.CTB6_CONVSEG
hanlp.pretrained.tok.LARGE_ALBERT_BASE
hanlp.pretrained.tok.PKU_NAME_MERGED_SIX_MONTHS_CONVSEG
hanlp.pretrained.tok.SIGHAN2005_MSR_CONVSEG
hanlp.pretrained.tok.SIGHAN2005_PKU_CONVSEG
Code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate the problem.
NER_1 = hanlp.pipeline() \
.append(hanlp.utils.rules.split_sentence, output_key='sentences') \
.append(hanlp.load('UD_TOK_MMINILMV2L12'), output_key='tok') \
.append(hanlp.load('CONLL03_NER_BERT_BASE_CASED_EN'), output_key='ner', input_key='tok')Describe the current behavior
A clear and concise description of what happened.
Expected behavior
A clear and concise description of what you expected to happen.
System information
- OS: Windows-10-10.0.19041-SP0
- Python: 3.9.0
- PyTorch: 1.13.1+cpu
- HanLP version: hanlp==2.1.0b45
Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
Decompressing C:\Users\AppData\Roaming\hanlp\ner/ner_conll03_bert_base_cased_en_20211227_121443.zip to C:\Users\AppData\Roaming\hanlp\ner
Failed to load https://file.hankcs.com/hanlp/ner/ner_conll03_bert_base_cased_en_20211227_121443.zip
If the problem still persists, please submit an issue to https://github.com/hankcs/HanLP/issues
When reporting an issue, make sure to paste the FULL ERROR LOG below.
......
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
1. Downgrade the protobuf package to 3.20.x or lower.
2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
- I've completed this form and searched the web for solutions.
已經跟著步驟執行還是一樣的報錯了，請問有別的方法嗎?
pip install -U hanlp[full]
Requirement already satisfied: hanlp[full] in c:\users\appdata\local\programs\python\python39\lib\site-packages (2.1.0b45)
Requirement already satisfied: hanlp-downloader in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (0.0.25)
Requirement already satisfied: pynvml in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (11.5.0)
Requirement already satisfied: toposort==1.5 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (1.5)
Requirement already satisfied: sentencepiece>=0.1.91 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (0.1.97)
Requirement already satisfied: hanlp-common>=0.0.19 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (0.0.19)
Requirement already satisfied: termcolor in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (1.1.0)
Requirement already satisfied: torch>=1.6.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (1.13.1)
Requirement already satisfied: hanlp-trie>=0.0.4 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (0.0.5)
Requirement already satisfied: transformers>=4.1.1 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (4.26.1)
Requirement already satisfied: tokenizers==0.11.6 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (0.11.6)
Requirement already satisfied: perin-parser>=0.0.12 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (0.0.12)
Requirement already satisfied: penman==1.2.1 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (1.2.1)
Requirement already satisfied: fasttext-wheel==0.9.2 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (0.9.2)
Requirement already satisfied: networkx>=2.5.1 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (3.0)
Requirement already satisfied: keras==2.6.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (2.6.0)
Requirement already satisfied: tensorflow==2.6.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (2.6.0)
Requirement already satisfied: pybind11>=2.2 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from fasttext-wheel==0.9.2->hanlp[full]) (2.10.3)
Requirement already satisfied: setuptools>=0.7.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from fasttext-wheel==0.9.2->hanlp[full]) (67.6.0)
Requirement already satisfied: numpy in c:\users\appdata\local\programs\python\python39\lib\site-packages (from fasttext-wheel==0.9.2->hanlp[full]) (1.19.5)
Requirement already satisfied: protobuf>=3.9.2 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (4.22.0)
Requirement already satisfied: astunparse~=1.6.3 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (1.6.3)
Requirement already satisfied: flatbuffers~=1.12.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (1.12)
Requirement already satisfied: absl-py~=0.10 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (0.15.0)
Requirement already satisfied: gast==0.4.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (0.4.0)
Requirement already satisfied: wrapt~=1.12.1 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (1.12.1)
Requirement already satisfied: typing-extensions~=3.7.4 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (3.7.4.3)
Requirement already satisfied: clang~=5.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (5.0)
Requirement already satisfied: h5py~=3.1.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (3.1.0)
Requirement already satisfied: tensorflow-estimator~=2.6 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (2.11.0)
Requirement already satisfied: tensorboard~=2.6 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (2.12.0)
Requirement already satisfied: six~=1.15.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (1.15.0)
Requirement already satisfied: keras-preprocessing~=1.1.2 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (1.1.2)
Requirement already satisfied: opt-einsum~=3.3.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (3.3.0)
Requirement already satisfied: grpcio<2.0,>=1.37.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (1.51.3)
Requirement already satisfied: wheel~=0.35 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (0.38.4)
Requirement already satisfied: google-pasta~=0.2 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (0.2.0)
Requirement already satisfied: phrasetree in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp-common>=0.0.19->hanlp[full]) (0.0.8)
Requirement already satisfied: scipy in c:\users\appdata\local\programs\python\python39\lib\site-packages (from perin-parser>=0.0.12->hanlp[full]) (1.10.1)
Requirement already satisfied: tqdm>=4.27 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from transformers>=4.1.1->hanlp[full]) (4.65.0)
Requirement already satisfied: filelock in c:\users\appdata\local\programs\python\python39\lib\site-packages (from transformers>=4.1.1->hanlp[full]) (3.9.0)
Requirement already satisfied: huggingface-hub<1.0,>=0.11.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from transformers>=4.1.1->hanlp[full]) (0.12.1)
Requirement already satisfied: regex!=2019.12.17 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from transformers>=4.1.1->hanlp[full]) (2022.10.31)
Requirement already satisfied: pyyaml>=5.1 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from transformers>=4.1.1->hanlp[full]) (6.0)
Requirement already satisfied: requests in c:\users\appdata\local\programs\python\python39\lib\site-packages (from transformers>=4.1.1->hanlp[full]) (2.28.2)
Requirement already satisfied: packaging>=20.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from transformers>=4.1.1->hanlp[full]) (23.0)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (0.4.6)
Requirement already satisfied: markdown>=2.6.8 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (3.4.1)
Requirement already satisfied: google-auth<3,>=1.6.3 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (2.16.2)
Requirement already satisfied: werkzeug>=1.0.1 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (2.2.3)
Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (0.7.0)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (1.8.1)
Requirement already satisfied: idna<4,>=2.5 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from requests->transformers>=4.1.1->hanlp[full]) (3.4)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from requests->transformers>=4.1.1->hanlp[full]) (2022.12.7)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from requests->transformers>=4.1.1->hanlp[full]) (3.1.0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from requests->transformers>=4.1.1->hanlp[full]) (1.26.14)
Requirement already satisfied: colorama in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tqdm>=4.27->transformers>=4.1.1->hanlp[full]) (0.4.6)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from google-auth<3,>=1.6.3->tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (5.3.0)
Requirement already satisfied: pyasn1-modules>=0.2.1 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from google-auth<3,>=1.6.3->tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (0.2.8)
Requirement already satisfied: rsa<5,>=3.1.4 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from google-auth<3,>=1.6.3->tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (4.9)
Requirement already satisfied: requests-oauthlib>=0.7.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (1.3.1)
Requirement already satisfied: importlib-metadata>=4.4 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from markdown>=2.6.8->tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (6.0.0)
Requirement already satisfied: MarkupSafe>=2.1.1 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from werkzeug>=1.0.1->tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (2.1.2)
Requirement already satisfied: zipp>=0.5 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (3.15.0)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (0.4.8)
Requirement already satisfied: oauthlib>=3.0.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (3.2.2)
Python 3.9.0 (tags/v3.9.0:9cf6752, Oct 5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import hanlp
>>> ner = hanlp.load(hanlp.pretrained.ner.CONLL03_NER_BERT_BASE_CASED_EN)
Failed to load https://file.hankcs.com/hanlp/ner/ner_conll03_bert_base_cased_en_20211227_121443.zip
If the problem still persists, please submit an issue to https://github.com/hankcs/HanLP/issues
When reporting an issue, make sure to paste the FULL ERROR LOG below.
================================ERROR LOG BEGINS================================
OS: Windows-10-10.0.19041-SP0
Python: 3.9.0
PyTorch: 1.13.1+cpu
Traceback (most recent call last):
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\hanlp\utils\component_util.py", line 99, in load_from_meta_file
obj: Component = object_from_classpath(cls)
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\hanlp_common\reflection.py", line 27, in object_from_classpath
classpath = str_to_type(classpath)
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\hanlp_common\reflection.py", line 44, in str_to_type
cls = getattr(importlib.import_module(module_name), class_name)
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 790, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\hanlp\components\ner\ner_tf.py", line 7, in <module>
import tensorflow as tf
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\__init__.py", line 41, in <module>
from tensorflow.python.tools import module_util as _module_util
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\__init__.py", line 40, in <module>
from tensorflow.python.eager import context
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\eager\context.py", line 32, in <module>
from tensorflow.core.framework import function_pb2
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\core\framework\function_pb2.py", line 16, in <module>
from tensorflow.core.framework import attr_value_pb2 as tensorflow_dot_core_dot_framework_dot_attr__value__pb2
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\core\framework\attr_value_pb2.py", line 16, in <module>
from tensorflow.core.framework import tensor_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__pb2
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\core\framework\tensor_pb2.py", line 16, in <module>
from tensorflow.core.framework import resource_handle_pb2 as tensorflow_dot_core_dot_framework_dot_resource__handle__pb2
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\core\framework\resource_handle_pb2.py", line 16, in <module>
from tensorflow.core.framework import tensor_shape_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__shape__pb2
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\core\framework\tensor_shape_pb2.py", line 36, in <module>
_descriptor.FieldDescriptor(
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\google\protobuf\descriptor.py", line 561, in __new__
_message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
1. Downgrade the protobuf package to 3.20.x or lower.
2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\hanlp\__init__.py", line 43, in load
return load_from_meta_file(save_dir, 'meta.json', verbose=verbose, **kwargs)
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\hanlp\utils\component_util.py", line 165, in load_from_meta_file
import tensorflow
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\__init__.py", line 41, in <module>
from tensorflow.python.tools import module_util as _module_util
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\__init__.py", line 40, in <module>
from tensorflow.python.eager import context
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\python\eager\context.py", line 32, in <module>
from tensorflow.core.framework import function_pb2
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\core\framework\function_pb2.py", line 16, in <module>
from tensorflow.core.framework import attr_value_pb2 as tensorflow_dot_core_dot_framework_dot_attr__value__pb2
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\core\framework\attr_value_pb2.py", line 16, in <module>
from tensorflow.core.framework import tensor_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__pb2
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\core\framework\tensor_pb2.py", line 16, in <module>
from tensorflow.core.framework import resource_handle_pb2 as tensorflow_dot_core_dot_framework_dot_resource__handle__pb2
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\core\framework\resource_handle_pb2.py", line 16, in <module>
from tensorflow.core.framework import tensor_shape_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__shape__pb2
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\tensorflow\core\framework\tensor_shape_pb2.py", line 36, in <module>
_descriptor.FieldDescriptor(
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\google\protobuf\descriptor.py", line 561, in __new__
_message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
1. Downgrade the protobuf package to 3.20.x or lower.
2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates其实log中Google已经有提示了：
Downgrade the protobuf package to 3.20.x or lower.
试试：
pip install protobuf==3.19.6
試過了可以了 不過為什麼這個protobuf的依賴沒有放進去Full的安裝裡了?
還有我能從哪裡看到那一個模型是需要完整版HanLP才能用?
(我剛剛開始學習用NLP，很多可能都不太懂)
protobuf是tensorflow的dependency，按道理应该由tensorflow负责安装。然而tensorflow兼容性做得实在是太差劲了，没有把自己的dependency管理好。
你可以从 https://hanlp.hankcs.com/docs/install.html#install-native-package 找到full的文档，具体哪个模型需要tf只能加载后查看它的type是否以TF结尾了。
你好 我看到了新加了依賴要protobuf<3.19 但是我試過成功的是用3.19.6的版本 現在要求<3.19好像有依賴沖突跑不起來了
pip install protobuf==3.19.6
pip install -U hanlp[full]后有问题？日志？
Google的protobuf 3.19在mac上不兼容，测试得出3.18才兼容。另外3.18也兼容Linux。
安裝
C:\Users\chancheh>pip install hanlp[full]
WARNING: Ignoring invalid distribution -rotobuf (c:\users\appdata\local\programs\python\python39\lib\site-packages)
WARNING: Ignoring invalid distribution - (c:\users\appdata\local\programs\python\python39\lib\site-packages)
WARNING: Ignoring invalid distribution -rotobuf (c:\users\appdata\local\programs\python\python39\lib\site-packages)
WARNING: Ignoring invalid distribution - (c:\users\appdata\local\programs\python\python39\lib\site-packages)
Requirement already satisfied: hanlp[full] in c:\users\appdata\local\programs\python\python39\lib\site-packages (2.1.0b46)
Requirement already satisfied: hanlp-common>=0.0.19 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (0.0.19)
Requirement already satisfied: hanlp-downloader in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (0.0.25)
Requirement already satisfied: torch>=1.6.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (1.13.1)
Requirement already satisfied: termcolor in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (1.1.0)
Requirement already satisfied: transformers>=4.1.1 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (4.26.1)
Requirement already satisfied: pynvml in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (11.5.0)
Requirement already satisfied: toposort==1.5 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (1.5)
Requirement already satisfied: hanlp-trie>=0.0.4 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (0.0.5)
Requirement already satisfied: tokenizers==0.11.6 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (0.11.6)
Requirement already satisfied: sentencepiece>=0.1.91 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (0.1.97)
Requirement already satisfied: networkx>=2.5.1 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (3.0)
Requirement already satisfied: perin-parser>=0.0.12 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (0.0.12)
Requirement already satisfied: keras==2.6.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (2.6.0)
Requirement already satisfied: fasttext-wheel==0.9.2 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (0.9.2)
Requirement already satisfied: protobuf<3.19 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (3.18.3)
Requirement already satisfied: penman==1.2.1 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (1.2.1)
Requirement already satisfied: tensorflow==2.6.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp[full]) (2.6.0)
Requirement already satisfied: numpy in c:\users\appdata\local\programs\python\python39\lib\site-packages (from fasttext-wheel==0.9.2->hanlp[full]) (1.19.5)
Requirement already satisfied: pybind11>=2.2 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from fasttext-wheel==0.9.2->hanlp[full]) (2.10.3)
Requirement already satisfied: setuptools>=0.7.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from fasttext-wheel==0.9.2->hanlp[full]) (67.6.0)
Requirement already satisfied: absl-py~=0.10 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (0.15.0)
Requirement already satisfied: google-pasta~=0.2 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (0.2.0)
Requirement already satisfied: h5py~=3.1.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (3.1.0)
Requirement already satisfied: gast==0.4.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (0.4.0)
Requirement already satisfied: grpcio<2.0,>=1.37.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (1.51.3)
Requirement already satisfied: flatbuffers~=1.12.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (1.12)
Requirement already satisfied: tensorboard~=2.6 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (2.11.2)
Requirement already satisfied: tensorflow-estimator~=2.6 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (2.11.0)
Requirement already satisfied: six~=1.15.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (1.15.0)
Requirement already satisfied: wrapt~=1.12.1 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (1.12.1)
Requirement already satisfied: clang~=5.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (5.0)
Requirement already satisfied: keras-preprocessing~=1.1.2 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (1.1.2)
Requirement already satisfied: typing-extensions~=3.7.4 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (3.7.4.3)
Requirement already satisfied: astunparse~=1.6.3 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (1.6.3)
Requirement already satisfied: wheel~=0.35 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (0.38.4)
Requirement already satisfied: opt-einsum~=3.3.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorflow==2.6.0->hanlp[full]) (3.3.0)
Requirement already satisfied: phrasetree in c:\users\appdata\local\programs\python\python39\lib\site-packages (from hanlp-common>=0.0.19->hanlp[full]) (0.0.8)
Requirement already satisfied: scipy in c:\users\appdata\local\programs\python\python39\lib\site-packages (from perin-parser>=0.0.12->hanlp[full]) (1.10.1)
Requirement already satisfied: regex!=2019.12.17 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from transformers>=4.1.1->hanlp[full]) (2022.10.31)
Requirement already satisfied: pyyaml>=5.1 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from transformers>=4.1.1->hanlp[full]) (6.0)
Requirement already satisfied: filelock in c:\users\appdata\local\programs\python\python39\lib\site-packages (from transformers>=4.1.1->hanlp[full]) (3.9.0)
Requirement already satisfied: huggingface-hub<1.0,>=0.11.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from transformers>=4.1.1->hanlp[full]) (0.12.1)
Requirement already satisfied: requests in c:\users\appdata\local\programs\python\python39\lib\site-packages (from transformers>=4.1.1->hanlp[full]) (2.28.2)
Requirement already satisfied: tqdm>=4.27 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from transformers>=4.1.1->hanlp[full]) (4.65.0)
Requirement already satisfied: packaging>=20.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from transformers>=4.1.1->hanlp[full]) (23.0)
Requirement already satisfied: werkzeug>=1.0.1 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (2.2.3)
Requirement already satisfied: google-auth<3,>=1.6.3 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (2.16.2)
Requirement already satisfied: markdown>=2.6.8 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (3.4.1)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (1.8.1)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (0.4.6)
Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (0.6.1)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from requests->transformers>=4.1.1->hanlp[full]) (3.1.0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from requests->transformers>=4.1.1->hanlp[full]) (1.26.14)
Requirement already satisfied: idna<4,>=2.5 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from requests->transformers>=4.1.1->hanlp[full]) (3.4)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from requests->transformers>=4.1.1->hanlp[full]) (2022.12.7)
Requirement already satisfied: colorama in c:\users\appdata\local\programs\python\python39\lib\site-packages (from tqdm>=4.27->transformers>=4.1.1->hanlp[full]) (0.4.6)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from google-auth<3,>=1.6.3->tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (5.3.0)
Requirement already satisfied: pyasn1-modules>=0.2.1 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from google-auth<3,>=1.6.3->tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (0.2.8)
Requirement already satisfied: rsa<5,>=3.1.4 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from google-auth<3,>=1.6.3->tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (4.9)
Requirement already satisfied: requests-oauthlib>=0.7.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (1.3.1)
Requirement already satisfied: importlib-metadata>=4.4 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from markdown>=2.6.8->tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (6.0.0)
Requirement already satisfied: MarkupSafe>=2.1.1 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from werkzeug>=1.0.1->tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (2.1.2)
Requirement already satisfied: zipp>=0.5 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (3.15.0)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (0.4.8)
Requirement already satisfied: oauthlib>=3.0.0 in c:\users\appdata\local\programs\python\python39\lib\site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.6->tensorflow==2.6.0->hanlp[full]) (3.2.2)
WARNING: Ignoring invalid distribution -rotobuf (c:\users\appdata\local\programs\python\python39\lib\site-packages)
WARNING: Ignoring invalid distribution - (c:\users\appdata\local\programs\python\python39\lib\site-packages)
WARNING: Ignoring invalid distribution -rotobuf (c:\users\appdata\local\programs\python\python39\lib\site-packages)
WARNING: Ignoring invalid distribution - (c:\users\appdata\local\programs\python\python39\lib\site-packages)
WARNING: Ignoring invalid distribution -rotobuf (c:\users\appdata\local\programs\python\python39\lib\site-packages)
WARNING: Ignoring invalid distribution - (c:\users\appdata\local\programs\python\python39\lib\site-packages)
WARNING: Ignoring invalid distribution -rotobuf (c:\users\appdata\local\programs\python\python39\lib\site-packages)
WARNING: Ignoring invalid distribution - (c:\users\appdata\local\programs\python\python39\lib\site-packages)
在pip freeze裡面看過了 hanlp==2.1.0b46
然後根據你這裡的碼跑了
你没有正确安装，请follow：https://colab.research.google.com/drive/1d95nzqyscZchsmZ7_GQ15J3LGWqvPjpr?usp=sharing
Python 3.9.0 (tags/v3.9.0:9cf6752, Oct 5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import hanlp
>>> ner = hanlp.load(hanlp.pretrained.ner.CONLL03_NER_BERT_BASE_CASED_EN)
Failed to load https://file.hankcs.com/hanlp/ner/ner_conll03_bert_base_cased_en_20211227_121443.zip
If the problem still persists, please submit an issue to https://github.com/hankcs/HanLP/issues
When reporting an issue, make sure to paste the FULL ERROR LOG below.
================================ERROR LOG BEGINS================================
OS: Windows-10-10.0.19041-SP0
Python: 3.9.0
PyTorch: 1.13.1+cpu
TensorFlow cannot be imported due to ImportError: cannot import name 'builder' from 'google.protobuf.internal' (C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\google\protobuf\internal\__init__.py). Note this is not a bug of HanLP, but rather a compatability issue caused by TensorFlow.
HanLP: 2.1.0-beta.46
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\hanlp\__init__.py", line 43, in load
return load_from_meta_file(save_dir, 'meta.json', verbose=verbose, **kwargs)
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\hanlp\utils\component_util.py", line 186, in load_from_meta_file
raise e from None
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\hanlp\utils\component_util.py", line 99, in load_from_meta_file
obj: Component = object_from_classpath(cls)
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\hanlp_common\reflection.py", line 27, in object_from_classpath
classpath = str_to_type(classpath)
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\hanlp_common\reflection.py", line 44, in str_to_type
cls = getattr(importlib.import_module(module_name), class_name)
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 790, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\hanlp\components\ner\ner_tf.py", line 7, in <module>
import tensorflow as tf
File "C:\Users\AppData\Roaming\Python\Python39\site-packages\tensorflow\__init__.py", line 38, in <module>
from tensorflow.python.tools import module_util as _module_util
File "C:\Users\AppData\Roaming\Python\Python39\site-packages\tensorflow\python\__init__.py", line 37, in <module>
from tensorflow.python.eager import context
File "C:\Users\AppData\Roaming\Python\Python39\site-packages\tensorflow\python\eager\context.py", line 28, in <module>
from tensorflow.core.framework import function_pb2
File "C:\Users\AppData\Roaming\Python\Python39\site-packages\tensorflow\core\framework\function_pb2.py", line 5, in <module>
from google.protobuf.internal import builder as _builder
ImportError: cannot import name 'builder' from 'google.protobuf.internal' (C:\Users\AppData\Local\Programs\Python\Python39\lib\site-packages\google\protobuf\internal\__init__.py)
>>> print(ner(["President", "Obama", "is", "speaking", "at", "the", "White", "House", "."]))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'ner' is not defined你的log已经告诉你答案了：
WARNING: Ignoring invalid distribution -rotobuf (c:\users\appdata\local\programs\python\python39\lib\site-packages)
这说明你的Windows损坏了protobuf。建议卸载Python后删除c:\users\appdata\local\programs\python\python39\lib\site-packages再重新安装Python，或者使用Linux或mac。
根据unit test，HanLP在Windows Python3.9下没有问题：
https://github.com/hankcs/HanLP/actions/runs/4387038948/jobs/7681840330