Koichi Akabe's starred repositories
python-vaporetto
🛥 Vaporetto is a fast and lightweight pointwise prediction based tokenizer. This is a Python wrapper for Vaporetto.
self-description-set
A formula that become itself when plotted
python-daachorse
🐎 A fast implementation of the Aho-Corasick algorithm using the compact double-array data structure. (Python wrapper for daachorse)
include-bytes-zstd
Includes a file with zstd compression in Rust
rust-lbfgs
LBFGS optimization algorithm ported from liblbfgs
aho-corasick
🌿 A simple Node.js wrapper for Rust's native implementation.
tokenizer-speed-bench
Comparison code of various tokenizers