Giters
BBuf
/
RWKV-World-HF-Tokenizer
Geek Repo:
Geek Repo
Github PK Tool:
Github PK Tool
Stargazers:
33
Watchers:
4
Issues:
3
Forks:
5
BBuf/RWKV-World-HF-Tokenizer Issues
The tokenizer is 2.5x slower than other huggingface tokenizer and the original blinks world tokenizer
Updated
4 months ago
how to create tokenizer.json
Updated
5 months ago
Comments count
4
Is tokenization_rwkv5.py equivalent to tokenization_rwkv_world.py from huggingface?
Updated
5 months ago
Comments count
2