crabml is an ongoing experiment that aims to reimplement GGML using Rust.
Currently it can inference a 3B Q8_0 quantized Llama model at a dog slow speed.
Its design goals are:
- focus on inference only.
- limit tensor operators to the bare minimum required for LLM inference.
- fast enough inferencing on cheap hardwares.
mmap()from day one.- prioritize SIMD ahead of GPU.
RUSTFLAGS="-C target-feature=+neon" cargo build --release
./target/release/crabml-cli -m ./testdata/open-llama-3b-q8_0.gguf "captain america" --steps 100 -t 0.8 -p 1.0