rinnakk / japanese-pretrained-models

Code for producing Japanese pretrained models provided by rinna Co., Ltd.

Home Page:https://huggingface.co/rinna

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Please add "tokenizer_class" in "config.json"

shirayu opened this issue · comments

Please add tokenizer_class in config.json like

  "tokenizer_class": "T5Tokenizer",

.
This enables use of AutoTokenizer like

tokenizer = AutoTokenizer.from_pretrained("rinna/japanese-gpt-1b")

instead of

tokenizer = T5Tokenizer.from_pretrained("rinna/japanese-gpt-1b")

(Other models can be changed in the same way.)

Related to: cl-tohoku/bert-japanese#24

Thank you. It is now solved in this commit.