ModuleNotFoundError: No module named 'ipex_llm.vllm.xpu' while using docker and installation
sachinchandra opened this issue · comments
followed installation of Vllm via this link
tried running via docker too: Here is the image
Still getting the no module named ipex_llm.vllm.xpu
Using currently on Arc A770
python -m ipex_llm.vllm.xpu.entrypoints.openai.api_server \
--served-model-name $served_model_name \
--port 8000 \
--model $model \
--trust-remote-code \
--gpu-memory-utilization 0.75 \
--device xpu \
--dtype float16 \
--enforce-eager \
--load-in-low-bit sym_int4 \
--max-model-len 4096 \
--max-num-batched-tokens 10240 \
--max-num-seqs 12 \
--tensor-parallel-size 2
``
/usr/local/lib/python3.11/dist-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: ''If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source?
warn(
[2024-05-24 19:06:41,339] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to xpu (auto detect)
[WARNING] async_io requires the dev libaio .so object and headers but these were not found.
[WARNING] async_io: please install the libaio-dev package with apt
[WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
2024-05-24 19:06:41,508 - root - INFO - intel_extension_for_pytorch auto imported
/usr/bin/python: Error while finding module specification for 'ipex_llm.vllm.xpu.entrypoints.openai.api_server' (ModuleNotFoundError: No module named 'ipex_llm.vllm.xpu')
We refactored the vLLM code today, the new code and image are in the CI/CD progress, while the doc is updated first.
Please try python -m ipex_llm.vllm.entrypoints.openai.api_server
for today's ipex-llm build.
And with the nightly build tomorrow, please follow the doc.