Giters

intel-analytics / ipex-llm

Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, Baichuan, Mixtral, Gemma, Phi, etc.) on Intel CPU and GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, DeepSpeed, vLLM, FastChat, Axolotl, etc.

Home Page:https://ipex-llm.readthedocs.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

LLaVA import error on CPU

tsantra opened this issue · comments

commented

getting an import error while using LLaVA model.

command:
python generate.py --repo-id-or-model-path liuhaotian/llava-v1.6-34b --image-path-or-url 'https://llava-vl.github.io/static/images/monalisa.jpg'

image

the LLaVA directory has:
image

env check:

PYTHON_VERSION=3.11.9

transformers=4.36.2

torch=2.1.2+cpu

ipex-llm Version: 2.1.0b20240605

IPEX is not installed.

CPU Information:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 52 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 128
On-line CPU(s) list: 0-127
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) Gold 6430L
CPU family: 6
Model: 143
Thread(s) per core: 2
Core(s) per socket: 32
Socket(s): 2
Stepping: 7
CPU max MHz: 3400.0000
CPU min MHz: 800.0000
BogoMIPS: 3800.00

Total CPU Memory: 125.489 GB

Operating System:
Ubuntu 22.04.4 LTS \n \l

Linux ceed-user 5.15.0-107-generic #117-Ubuntu SMP Fri Apr 26 12:26:49 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

CLI:
Version: 1.2.22.20231126
Build ID: 00000000

Service:
Version: 1.2.22.20231126
Build ID: 00000000
Level Zero Version: 1.11.0

Driver UUID 32332e32-322e-3236-3531-362e33340000
Driver Version 23.22.26516.34
Driver Version 2023.16.12.0.12_195853.xmain-hotfix
Driver Version 2023.16.12.0.12_195853.xmain-hotfix

Driver related package version:
ii intel-fw-gpu 2023.39.2-25522.04 all Firmware package for Intel integrated and discrete GPUs
ii intel-i915-dkms 1.23.6.42.230425.56+i81-1 all Out of tree i915 driver.
ii intel-level-zero-gpu 1.3.26516.34-68222.04 amd64 Intel(R) Graphics Compute Runtime for oneAPI Level Zero.
ii level-zero-dev 1.11.0-649~22.04 amd64 Intel(R) Graphics Compute Runtime for oneAPI Level Zero.

env-check.sh: line 167: sycl-ls: command not found
igpu not detected

xpu-smi is properly installed.

+-----------+--------------------------------------------------------------------------------------+
| Device ID | Device Information |
+-----------+--------------------------------------------------------------------------------------+
| 0 | Device Name: Intel(R) Data Center GPU Flex 170 |
| | Vendor Name: Intel(R) Corporation |
| | SOC UUID: 00000000-0000-0000-6cf6-5109f1c50433 |
| | PCI BDF Address: 0000:ae:00.0 |
| | DRM Device: /dev/dri/card1 |
| | Function Type: physical |
+-----------+--------------------------------------------------------------------------------------+
GPU0 Memory size=16M

02:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 52) (prog-if 00 [VGA controller])
DeviceName: Onboard VGA
Subsystem: ASPEED Technology, Inc. ASPEED Graphics Family
Flags: bus master, medium devsel, latency 0, IRQ 16, NUMA node 0
Memory at 94000000 (32-bit, non-prefetchable) [size=16M]
Memory at 95140000 (32-bit, non-prefetchable) [size=256K]
I/O ports at 2000 [size=128]
Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
Capabilities:
Kernel driver in use: ast

commented

@jenniew Take a look at this?

commented

could you please let me know if there is any update on this.

commented

Sure, we're looking at this issue.

commented

Hi @jenniew , any update on this issue? We are working on a solution with a customer use case and any update will be immensely helpful. It is blocking us now.

commented

Hi, @tsantra!

We are now fixing the LLaVA example on CPU. However, you could try running LLaVA again using CPU on Linux with following installation steps:

conda create -n llm python=3.11 # recommend to use Python 3.11
conda activate llm

# install the latest ipex-llm nightly build with 'all' option
pip install --pre --upgrade ipex-llm[all] --extra-index-url https://download.pytorch.org/whl/cpu

git clone https://github.com/haotian-liu/LLaVA.git # clone the llava libary
cd LLaVA # change the working directory to the LLaVA folder
git checkout tags/v1.2.0 -b 1.2.0 # Get the branch which is compatible with transformers 4.36
pip install -e .
cd ..

We have tested that these steps works fine with latest ipex-llm when running LLaVA using CPU on linux. Please feel free to ask if there's any problem : )

commented

Hi @JinBridger

These steps work for me. Thank you.

However, I saved the model in 4bit and tried to load the saved 4bit model, when I am getting an error.
I opened another ticket for that:
#11342

Could you please advise on what to do?