Convert Tokenizer to a Class
ConProgramming opened this issue · comments
Love this project, solid work! Would be better DX and more maintainable codebase if LlamaTokenizer was a class though!🙏
belladoreai / llama-tokenizer-js
JS tokenizer for LLaMA 1 and 2
https://belladoreai.github.io/llama-tokenizer-js/example-demo/build/
