simonw / ttok

Count and truncate text based on tokens

simonw/ttok Issues

Options for special token handling
Closed 4 months ago3
Does `ttok` need an internet connection?
Updated 4 months ago1
Ability to count tokens for models other than OpenAI
Updated 5 months ago2
ttok adds a 20 second delay when using llm-gpt4all when offline
Updated 9 months ago1
Mechanism for splitting rather than truncating
Updated 9 months ago21
An additional option to view the actual tokenized text
Closed a year ago11
Option to turn token integers back into text
Closed a year ago6
Truncate => trim tokens
Updated a year ago
Verbose option
Updated a year ago
Initial design
Closed a year ago3