philippgille / tokenizer

Pure Go implementation of OpenAI's tiktoken tokenizer

Geek Repo

Github PK Tool