belladoreai / llama-tokenizer-js

JS tokenizer for LLaMA 1 and 2

https://belladoreai.github.io/llama-tokenizer-js/example-demo/build/

belladoreai/llama-tokenizer-js Issues

CommonJS
Updated 7 months ago3
Typescript support
Updated 8 months ago5
Code Llama support
Closed 8 months ago2
Integration into 🤗 Transformers.js (+ generalization/minor improvements)
Closed 10 months ago5
`performance` not available
Closed a year ago2
Contributions
Closed a year ago2
CJK language
Closed a year ago5
Colors
Closed a year ago5
Discussion: using this with sqlite-vss
Closed a year ago3
Convert Tokenizer to a Class
Updated a year ago