AquilaChat2 34B推理问题

[宁鹏涛]

System Info

目前发现使用transformers推理报错，以下是错误信息。

Traceback (most recent call last):
File "/home/ai/projects/FlagAI/predict.py", line 6, in
tokenizer = AutoTokenizer.from_pretrained(model_info, trust_remote_code=True)
File "/home/ai/miniconda3/envs/flagai/lib/python3.9/site-packages/transformers/models/auto/tokenization_auto.py", line 751, in from_pretrained
return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
File "/home/ai/miniconda3/envs/flagai/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 2045, in from_pretrained
return cls._from_pretrained(
File "/home/ai/miniconda3/envs/flagai/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 2256, in _from_pretrained
tokenizer = cls(*init_inputs, **init_kwargs)
File "/home/ai/miniconda3/envs/flagai/lib/python3.9/site-packages/transformers/models/gpt2/tokenization_gpt2_fast.py", line 136, in init
super().init(
File "/home/ai/miniconda3/envs/flagai/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py", line 111, in init
fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
Exception: EOF while parsing a value at line 156568 column 3

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as T5/AltCLIP, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
device = torch.device("cuda")
#model_info = "BAAI/AquilaChat2-34B"
#本地模型
model_info = "/home/ai/projects/AquilaChat2-34B"
tokenizer = AutoTokenizer.from_pretrained(model_info, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_info, trust_remote_code=True)
model.eval()
model.to(device)
text = "请给出10个要到北京旅游的理由。"
tokens = tokenizer.encode_plus(text)['input_ids']
tokens = torch.tensor(tokens)[None,].to(device)
stop_tokens = ["###", "[UNK]", ""]
with torch.no_grad():
out = model.generate(tokens, do_sample=True, max_length=512, eos_token_id=100007, bad_words_ids=[[tokenizer.encode(token)[0] for token in stop_tokens]])[0]
out = tokenizer.decode(out.cpu().numpy().tolist())
print(out)

Expected behavior

正常加载tokenizer

[huyz-git]

huggingface 上的 tokenizer.json 是坏的，model.baai.ac.cn 上的是好的。