dmitry-brazhenko / SharpToken

SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.

https://www.nuget.org/packages/SharpToken

dmitry-brazhenko/SharpToken Issues

o200k_base support
Closed 3 months ago1
Gbt4o support
Updated 3 months ago
cohere support
Updated 4 months ago
Performance compared to TiktokenSharp, Tokenizer
Closed 5 months ago7
Anthropic (claude) support
Updated 5 months ago5
GptEncoding thread-safe?
Closed 6 months ago4
New embedding models not recognized
Closed 6 months ago11
Implement MODEL_PREFIX_FOR_ENCODING
Closed a year ago2
gpt-3.5-turbo-16k Not Supported
Closed a year ago1
Missing model name for 3.5 turbo Azure Deployment model
Closed a year ago5
outdated lib Newtonsoft
Closed a year ago1
Incorrect token count with Cyrillic
Closed a year ago2
Add Microsoft.SourceLink.GitHub to publish repo info/commit to nuget.org
Closed a year ago1