[Bug] AttributeError: module 'tokenizers.decoders' has no attribute 'Replace'
jiaoyang3 opened this issue · comments
Describe the bug
/Users/lambert/anaconda3/bin/python /Users/lambert/PycharmProjects/pythonProject/InternML/demo3.py
Ignored unknown kwarg option byte_fallback
Traceback (most recent call last):
File "/Users/lambert/PycharmProjects/pythonProject/InternML/demo3.py", line 3, in
tokenizer = AutoTokenizer.from_pretrained("internlm/internlm2-chat-7b", trust_remote_code=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/lambert/anaconda3/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 714, in from_pretrained
return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/lambert/anaconda3/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 1854, in from_pretrained
return cls._from_pretrained(
^^^^^^^^^^^^^^^^^^^^^
File "/Users/lambert/anaconda3/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2017, in _from_pretrained
tokenizer = cls(*init_inputs, **init_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/lambert/.cache/huggingface/modules/transformers_modules/internlm/internlm2-chat-7b/1334f6269ff81eedf855ab13e4bdbe3c3d7ab3f4/tokenization_internlm2_fast.py", line 131, in init
super().init(
File "/Users/lambert/anaconda3/lib/python3.11/site-packages/transformers/tokenization_utils_fast.py", line 114, in init
fast_tokenizer = convert_slow_tokenizer(slow_tokenizer)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/lambert/anaconda3/lib/python3.11/site-packages/transformers/convert_slow_tokenizer.py", line 1332, in convert_slow_tokenizer
return converter_class(transformer_tokenizer).converted()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/lambert/anaconda3/lib/python3.11/site-packages/transformers/convert_slow_tokenizer.py", line 540, in converted
tokenizer.decoder = self.decoder(replacement, add_prefix_space)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/lambert/.cache/huggingface/modules/transformers_modules/internlm/internlm2-chat-7b/1334f6269ff81eedf855ab13e4bdbe3c3d7ab3f4/tokenization_internlm2_fast.py", line 61, in decoder
decoders.Replace("▁", " "),
^^^^^^^^^^^^^^^^
AttributeError: module 'tokenizers.decoders' has no attribute 'Replace'
Process finished with exit code 1
Environment
Python 3.11.5
Other information
No response
What's the version of transformers? Updating transformers may works
What's the version of transformers? Updating transformers may works
the latest version
4.37.1
What's the version of tokenizers?
This issue is marked as stale because it has been marked as invalid or awaiting response for 7 days without any further response. It will be closed in 7 days if the stale label is not removed or if there is no further response.
This issue is closed because it has been stale for 7 days. Please open a new issue if you have similar issues or you have any new updates now.