Giters

intel / ipex-llm

Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, DeepSeek, Mixtral, Gemma, Phi, MiniCPM, Qwen-VL, MiniCPM-V, etc.) on Intel XPU (e.g., local PC with iGPU and NPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM, DeepSpeed, Axolotl, etc.

Repository from Github https://github.comintel/ipex-llmRepository from Github https://github.comintel/ipex-llm

Harness does not work properly

RobinJing opened this issue · comments

commented

Describe the bug
Wanna use Harness but does not work.
How to reproduce
Steps to reproduce the error:

  1. Using Conda:
  • conda create with python 3.11
  • conda activate& git clone harnesss
  • pip install -e .
    Image
    2.Using Docker b11:
  • pip install -e . successfully
  • python run_multi_llb.py --model ipex-llm --pretrained /model/DeepSeek-R1-Distill-Qwen-32B --precision sym_int4 --device xpu:0,1,2,3 --tasks mmlu --batch 1 --no_cache
    Image
  • pip install datasets==2.21.0
  • python run_multi_llb.py --model ipex-llm --pretrained /model/DeepSeek-R1-Distill-Qwen-32B --precision sym_int4 --device xpu:0,1,2,3 --tasks mmlu --batch 1 --no_cache
    Image
  • pip install accelerate==0.26.0 & python run_multi_llb.py again
    Image
    -pip install trl==0.11.0 & python run_multi_llb.py again

Image

commented

For the conda environment, I have setup the environment and the deployment is ready to go, after this , I can start the script with:
python run_multi_llb.py --model ipex-llm --pretrained /model/DeepSeek-R1-Distill-Qwen-32B --precision sym_int4 --device xpu:0,1,2,3 --tasks hellaswag --batch 1 --no_cache
but the program stops suddenly without any meaningful output:

Image

commented

Maybe you can try to run it again using latest b16 image, and we don't need to install conda again in docker container, you can use pip install directly to test.

commented

Hi,
I use b16 image docker to perform the test, simply I start the backend server from vllm, and use harness 'local-completions' as the frontend, when I run the winogrande task, it works fine; however if I run mmlu task, the vllm server crashes:

Image

commented

If you want to run multi-card for one single large model, refer to this guide: https://github.com/intel/ipex-llm/tree/main/python/llm/dev/benchmark/ceval#multi-gpu-environment

run_multi_llb.py is supposed to run multiple tasks across multi cards and one model & task on a single card.

commented

Hi, I use b16 image docker to perform the test, simply I start the backend server from vllm, and use harness 'local-completions' as the frontend, when I run the winogrande task, it works fine; however if I run mmlu task, the vllm server crashes:

Image

And this error looks like an OOM error, can you provide the entile steps on how to run vllm and how to reproduce this issue?

commented

Synced offline, may check the memory usage of each card to confirm if the issue is OOM and paste your environment/steps as Jian suggested.