How to train a custom tokenizer for Chinese from scratch

Question

SparkJiao opened this issue a year ago · comments

Hi, wonderful work!

May I know how to train a custom tokenizer for Chinese from scratch? Is there any public reference or code can share?

Thanks for your help very much!

best,
Fangkai