Tokenizer
Work in progress.
Installation from git requires rustc and cargo to be available and and the wasm32-wasi target to be installed.
General tokenizer library for the Web and Node. Supports Huggingface and Tiktoken formats
Work in progress.
Installation from git requires rustc and cargo to be available and and the wasm32-wasi target to be installed.
General tokenizer library for the Web and Node. Supports Huggingface and Tiktoken formats
BSD 2-Clause "Simplified" License