Please add "tokenizer_class" in "config.json"
shirayu opened this issue · comments
Yuta Hayashibe commented
Please add tokenizer_class
in config.json
like
"tokenizer_class": "T5Tokenizer",
.
This enables use of AutoTokenizer
like
tokenizer = AutoTokenizer.from_pretrained("rinna/japanese-gpt-1b")
instead of
tokenizer = T5Tokenizer.from_pretrained("rinna/japanese-gpt-1b")
(Other models can be changed in the same way.)
Related to: cl-tohoku/bert-japanese#24
Zhao Tianyu commented
Thank you. It is now solved in this commit.