cztomsik / ava

All-in-one desktop app for running LLMs locally.

Home Page:https://avapls.com

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

enable avx and avx2 if supported when GPU doesn't exist

prabirshrestha opened this issue · comments

Noticed that ollama recently added support for avx and avx2 instructions.

Fixed performance issues on Windows without GPU acceleration. Systems with AVX and AVX2 instruction sets should be 2-4x faster.

https://github.com/ollama/ollama/releases/tag/v0.1.27

According to this it might be just adding option to ./src/windows/create_zip.sh and/or compiling linux with similar flags

https://github.com/ggerganov/llama.cpp/blob/cb49e0f8c906e5da49e9f6d64a57742a9a241c6a/README.md?plain=1#L298

Superseded by #23

Closing in favor of #23