A project written in the Rust language with the goal of offline load of small LLM Model, specifically RAG (Retrieval Augmented Generation) on mobile devices.
Please downloaded openchat_3.5.Q4_K_M.gguf and tokenizer.json from Huggingface:
download TheBloke/openchat_3.5-GGUF openchat_3.5.Q4_K_M.gguf
download openchat/openchat_3.5 tokenizer.json
Copy them to the folder ./models/openchat-3.5-gguf/
and then
cargo run --release
A project written in the Rust language with the goal of offline load of small LLM Model, specifically RAG (Retrieval Augmented Generation) on mobile devices.
Apache License 2.0