syohex / tiktoken.el

tiktoken.el is a (not so) fast BPE tokeniser written in Emacs Lisp for use with OpenAI's models.

Geek Repo

Github PK Tool