[Feature Request] Provide IPEX-LLM as an executable to install in Windows

[Bibek KC]

Hello! Thank you for this awesome project. :)
I am trying to build one application that uses IPEX-LLM to run llm on Intel GPUs.
I want to do something like create an executable so that the non-tech users don't have to go through the hassle of installation.
For instance: Ollama provides executable for windows installation. I am expecting something similar.

So, is there some executable already available for ipex-llm somewhere? Or, can somebody give me any hints on creating one?
I want to serve llm using ollama on Intel GPUs in windows OS. So, I need an executable of ipex-llm with llama.cpp backend ipex-llm[cpp]

Thank you in advance for your help!

[SONG Ge]

Hi @bibekyess , you may see https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Quickstart/llama_cpp_quickstart.html to get the ipex-llm's support for llama.cpp.

[Bibek KC]

Hi @sgwhat! Thank you so much for your message.
I have gone through the documentation you provided before, but it didn't fit my requirements so I questioned. FYI, I want to create a service for users who should not go through these low level installations and should just install with one click.

Precisely, I want to make .exe file for easy installation/running of ipex-llm with llama.cpp in any windows PC, so can you clear if my understanding is correct?

After creating these llama.cpp’s executable files following the above documentations, if I copy all of them to another windows pc, and then also download ollama.exe and if I run them both together, is it supposed to work theoretically, provided the build-environment and test-environment is same?
Thanks!

[SONG Ge]

Hi @bibekyess ,

1. For your question1, you may extract the main/server executable files from the installed ipex-llm[cpp] and provide them directly to users for use.
2. For your question2, yes, these executable files can be copied to another windows pc and will run correctly.

[Bibek KC]

Ok great! Thank you so much for your response! :)