Run llama.cpp
from Deno
import { Llama } from "jsr:@divy/libllama";
import process from "node:process";
const engine = new Llama({
model: process.argv[2] || "./models/llama-2-7b-chat.Q2_K.gguf",
});
const text = process.argv[3];
engine.predict(text, {
tokenCallback: (token) => {
process.stdout.write(token);
return true;
},
});
Make sure to clone the repository with submodules and install prerequisites for building llama.cpp
.
Build deno-llama
using cmake
:
mkdir build
cd build
cmake ..
make
Run the example:
export LIBLLAMA_PATH=../build/libllama.dylib
deno run --allow-ffi example.ts \
./models/llama-2-7b-chat.Q2_K.gguf \
"What is the meaning of life?"
This project is licensed under the MIT License.
Thanks to the authors of llama.cpp
and go-llama
. A lot of the code in this repository is based on their work.