guillaume-be / rust-tokenizers

Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

guillaume-be/rust-tokenizers Stargazers