niieani / gpt-tokenizer

JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4. Port of OpenAI's tiktoken with additional features.

https://gpt-tokenizer.dev

niieani/gpt-tokenizer Issues

Dependency Dashboard
Updated 2 months ago
ChatMessage[] TypeScript warning
Updated 2 months ago2
Support function calls?
Updated 3 months ago2
avoid expensive initialization
Updated 3 months ago4
Picture tokens
Updated 4 months ago
the `isWithinTokenLimit` may be broken
Closed 5 months ago1
vercel/pkg Error with gpt-tokenizer
Updated 6 months ago
Is the algorithm of this repo consistent with openai tiktoken?
Updated 7 months ago
Error: Model name must be provided either during initialization or passed in to the method
Closed 8 months ago2
Huge memory consumsation of isWithinTokenLimit
Updated 8 months ago
How to set the model in encode / encode generator like encodeChat or encodeChatGenerator?
Updated 8 months ago
encodeChatGenerator undefined?
Closed 8 months ago16
Different calculating result between gpt-tokenizer and openai
Closed 8 months ago1
Error in NextJS edge function: Attempted import error: './index.js' does not contain a default export (imported as 'cjs')
Updated 10 months ago
isWithinTokenLimit fails due to lack of model being provided
Updated 10 months ago5
ReferenceError: Property 'TextDecoder' doesn't exist, js engine: hermes
Updated 10 months ago
Error: Invariant: Method expects to have requestAsyncStorage, none available
Updated 10 months ago
This module is not ready for CJK characters
Updated a year ago2
Improving documentation about the use of api vs importing function directly
Updated a year ago2
Tokens are slightly different from OpenAI Tokenizer
Closed a year ago1
Add support for `gpt-3.5-turbo-0613` and `gpt-3.5-turbo-16k-0613`
Updated a year ago
最近官方更新了GPT3.5-16K的模型，请问计算方式跟gpt-3.5-turbo是一样的吗
Updated a year ago
Calculated tokens much higher than actual
Closed a year ago10
comparison to other tokenizers
Closed a year ago2
Loading with script tags doesn't work
Closed a year ago1
How can I get prompts_tokens?
Closed a year ago3
gpt-tokenizer@2.0.0 token count is smaller than ChatGPT api response
Closed a year ago1
Which encoding is this using?
Closed a year ago3
CDN version is broken
Closed a year ago2
Import Issue on codebases with both module and common js files
Closed a year ago4