AttributeError: 'CodeGenTokenizer' object has no attribute 'encoder'. Did you mean: 'encode'?

[chixinran]

Version:

• python=3.10.9
• torch==2.0.1
• tokenizers==0.15.2 transformers==4.39.1
• sentence_transformers==2.3.1

Traceback

File "/userhome/cs2/clarachi/HKU-DASC7606-A2/eval_fewshot.py", line 277, in <module>
    main()
  File "/userhome/cs2/clarachi/HKU-DASC7606-A2/eval_fewshot.py", line 228, in main
    tokenizer, model = get_model(base_model=args.model)
  File "/userhome/cs2/clarachi/HKU-DASC7606-A2/eval_fewshot.py", line 166, in get_model
    tokenizer = CodeGenTokenizer.from_pretrained(base_model)
  File "/userhome/cs2/clarachi/miniconda3/envs/nlp_env/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2086, in from_pretrained
    return cls._from_pretrained(
  File "/userhome/cs2/clarachi/miniconda3/envs/nlp_env/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2325, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/userhome/cs2/clarachi/HKU-DASC7606-A2/tokenization_codegen.py", line 170, in __init__
    super().__init__(
  File "/userhome/cs2/clarachi/miniconda3/envs/nlp_env/lib/python3.10/site-packages/transformers/tokenization_utils.py", line 367, in __init__
    self._add_tokens(
  File "/userhome/cs2/clarachi/miniconda3/envs/nlp_env/lib/python3.10/site-packages/transformers/tokenization_utils.py", line 467, in _add_tokens
    current_vocab = self.get_vocab().copy()
  File "/userhome/cs2/clarachi/HKU-DASC7606-A2/tokenization_codegen.py", line 203, in get_vocab
    return dict(self.encoder, **self.added_tokens_encoder)
AttributeError: 'CodeGenTokenizer' object has no attribute 'encoder'. Did you mean: 'encode'?

Error

AttributeError: 'CodeGenTokenizer' object has no attribute 'encoder'. Did you mean: 'encode'?

Could anyone help me debug pls!! Thx!!