Giters

BBuf / run-rwkv-world-4-in-mlc-llm

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

the tokenizer

KhalidAlt opened this issue · comments

commented

Hi,

I would like to ask is there a way to load the tokenizer using Autotokenizer from Huggingface. Also I noticed that when I downloaed the tokenizer from this link, I did not find a file named "tokenizer_model.zip".

I tried to change the extension of the file from .tar.gz to .zip and extracted it to a folder named "tokenizer_model". However, I was unable to load the tokenizer using Huggingface.

Could you provide any guidance on this matter?

Thank you.

commented

The HuggingFace version of the RWKV "world" model tokenizer is currently under development and is not supported yet.