lenML / llama-tokenizer-playground

@lenml/llama2-tokenizer playground

⚠️ IMPORTANT UPDATE ⚠️

Our library @lenml/llama2-tokenizer has been deprecated. We are excited to introduce our new library @lenml/tokenizers as its replacement, offering a broader set of features and an enhanced experience.

Why switch to @lenml/tokenizers?

• Fully Compatible with transformers.js Interfaces: Seamlessly supports all interfaces defined in transformers.js, making migration and integration effortless.
• Support for a Wide Range of Models: Regardless of which model you need, our new library supports it, ensuring broader coverage.
• Rich Feature Implementation: Includes a complete implementation of chat templates and normalizers to better serve your text processing and tokenization needs.

check out lenML/tokenizers.

llama-tokenizer-playground

here is demo for llama2-tokenizer.js package

deploy on github pages

TODOs

• dark theme
• tokenizer comparer
• fix react-highlight-within-textarea text style (when paste) issue