Local build failing to run (NO AVX2)
Gnomesenpai opened this issue · comments
Following the guide for locally building as my test system does not have AVX2 - I am using dual E5-2637v2's.
root@lab:~/turbopilot/build/ggml/build/bin# ./codegen-serve -t 16 /root/turbopilot/models/codegen-6B-multi-ggml-4bit-quant.bin
Illegal instruction
Is it possible to compile without the need for AVX2?
Hi there, I'm working on shipping builds without AVX and AVX2 in the next release but if you are building it yourself locally you should be able to disable AVX 2 by passing -DGGML_AVX2=OFF at the cmake stage.
I.e. full command would be something like cmake -DGGML_AVX2=OFF ..
FWIW release 0.0.5 now contains separate AVX and AVX 2 builds :)
Re-opening as this still exists as an issue, i am running the AVX build, system has 128GB ram with plenty available.
root@lab:~/copilot# ./turbopilot -m wizardcoder -t 32 -f ./models/WizardCoder-15B-1.0.ggmlv3.q4_0.bin
[2023-09-07 22:10:08.248] [info] Initializing Starcoder/Wizardcoder type model for 'wizardcoder' model type
[2023-09-07 22:10:08.248] [info] Attempt to load model from wizardcoder
load_model: loading model from './models/WizardCoder-15B-1.0.ggmlv3.q4_0.bin'
load_model: n_vocab = 49153
load_model: n_ctx = 8192
load_model: n_embd = 6144
load_model: n_head = 48
load_model: n_layer = 40
load_model: ftype = 2002
load_model: qntvr = 2
load_model: ggml ctx size = 25608.48 MB
Illegal instruction