javirandor / anthropic-tokenizer

Approximation of the Claude 3 tokenizer by inspecting generation stream

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

javirandor/anthropic-tokenizer Stargazers