Giters

niieani / gpt-tokenizer

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT models (o1, o3, o4, gpt-4o, gpt-4, etc.). Port of OpenAI's tiktoken with additional features.

Home Page:https://gpt-tokenizer.dev

Repository from Github https://github.comniieani/gpt-tokenizerRepository from Github https://github.comniieani/gpt-tokenizer

niieani/gpt-tokenizer Stargazers