salesforce / CodeGen

CodeGen is an open-source model for program synthesis. Trained on TPU-v4. Competitive with OpenAI Codex.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

ValueError: Tokenizer class CodeGenTokenizer does not exist or is not currently imported.

robotPin opened this issue · comments

https://huggingface.co/Salesforce/codegen-2B-mono
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("Salesforce/codegen-2B-mono")
model = AutoModelForCausalLM.from_pretrained("Salesforce/codegen-2B-mono")

text = "def hello_world():"
input_ids = tokenizer(text, return_tensors="pt").input_ids

generated_ids = model.generate(input_ids, max_length=128)
print(tokenizer.decode(generated_ids[0], skip_special_tokens=True))

ValueError: Tokenizer class CodeGenTokenizer does not exist or is not currently imported.

@robotPin Could you let me know the transformers version? Our model is not included in a release yet, so you'd have to install from the repo, i.e.

pip install git+https://github.com/huggingface/transfomers.git

Version: 4.20.1

Upgrade to the Version: 4.21.0.dev0 problem solving

thank you @rooa