microsoft / Tokenizer

Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.

https://github.com/microsoft/Tokenizer/

microsoft/Tokenizer Issues

Error: ENOENT: no such file or directory, mkdir '/var/model'
Closed 2 months ago1
Support for text-embedding-3-large
Closed 2 months ago1
Tokenizer TS Web Browser Compatability
Updated 3 months ago1
TS Optimization: make the LRU cache pluggable and faster
Closed 4 months ago
TS Optimization: avoid allocations and array slicing in byte pair encoding
Closed 4 months ago
TS Optimization: avoid allocations when encoding text
Closed 4 months ago
TS Optimization: see if there's a minimum length for LRU entries
Closed 4 months ago1
`encodeTrim*` functions behave weirdly with some (special?) tokens
Closed 8 months ago
Token count accuracy questions with GPT3.5
Closed 10 months ago2
Can I use the SentencePiece Model file?
Closed a year ago1
Example of how to pre-download the BPE rank file
Closed a year ago2