AttributeError: module 'onnxruntime_genai' has no attribute 'Model'
iwaitu opened this issue · comments
When I run the phi3-qa.py I got some error:
(onnx) iwaitu@DESKTOP-RDS3VMA:~/work/onnx$ python phi3-qa.py -m cuda-int4-rtn-block-32
Traceback (most recent call last):
File "/mnt/d/WSLData/onnx/phi3-qa.py", line 96, in <module>
main(args)
File "/mnt/d/WSLData/onnx/phi3-qa.py", line 14, in main
model = og.Model(f'{args.model}')
AttributeError: module 'onnxruntime_genai' has no attribute 'Model'
??!!!
Hi @iwaitu, can you please list the packages that you have installed?
Hi @iwaitu any update here?
The error you're seeing is not expected specially since you're able to successfully import onnxruntime_genai if your code has reached that point.
@baijumeswani @natke I cant run it with cpu now. thx.
But I still can't run it with cuda.
import onnxruntime_genai as og
model = og.Model('cuda/cuda-int4-rtn-block-32')
tokenizer = og.Tokenizer(model)
tokenizer_stream = tokenizer.create_stream()
# Set the max length to something sensible by default,
# since otherwise it will be set to the entire context length
search_options = {}
search_options['max_length'] = 2048
chat_template = '<|user|>\n{input} <|end|>\n<|assistant|>'
text = input("Input: ")
if not text:
print("Error, input cannot be empty")
exit
prompt = f'{chat_template.format(input=text)}'
input_tokens = tokenizer.encode(prompt)
params = og.GeneratorParams(model)
params.set_search_options(**search_options)
params.input_ids = input_tokens
generator = og.Generator(model, params)
print("Output: ", end='', flush=True)
try:
while not generator.is_done():
generator.compute_logits()
generator.generate_next_token()
new_token = generator.get_next_tokens()[0]
print(tokenizer_stream.decode(new_token), end='', flush=True)
except KeyboardInterrupt:
print(" --control+c pressed, aborting generation--")
print()
del generator
I got this message:
Traceback (most recent call last):
File "/home/iwaitu/work/onnx/demo_cuda.py", line 3, in <module>
model = og.Model('cuda/cuda-int4-rtn-block-32')
onnxruntime_genai.onnxruntime_genai.OrtException: CUDA execution provider is not enabled in this build.
pip list
Package Version
------------------------ -----------
alembic 1.13.1
annotated-types 0.7.0
certifi 2024.2.2
charset-normalizer 3.3.2
coloredlogs 15.0.1
colorlog 6.8.2
filelock 3.14.0
flatbuffers 24.3.25
fsspec 2024.5.0
greenlet 3.0.3
huggingface-hub 0.23.2
humanfriendly 10.0
idna 3.7
Jinja2 3.1.4
lightning-utilities 0.11.2
Mako 1.3.5
MarkupSafe 2.1.5
mpmath 1.3.0
networkx 3.3
numpy 1.26.4
nvidia-cublas-cu12 12.1.3.1
nvidia-cuda-cupti-cu12 12.1.105
nvidia-cuda-nvrtc-cu12 12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12 8.9.2.26
nvidia-cufft-cu12 11.0.2.54
nvidia-curand-cu12 10.3.2.106
nvidia-cusolver-cu12 11.4.5.107
nvidia-cusparse-cu12 12.1.0.106
nvidia-nccl-cu12 2.20.5
nvidia-nvjitlink-cu12 12.5.40
nvidia-nvtx-cu12 12.1.105
olive-ai 0.6.1
onnx 1.16.1
onnxruntime-genai 0.3.0rc2
onnxruntime-genai-cuda 0.3.0rc2
onnxruntime-gpu 1.18.0
optuna 3.6.1
packaging 24.0
pandas 2.2.2
pip 24.0
protobuf 3.20.3
pydantic 2.7.2
pydantic_core 2.18.3
python-dateutil 2.9.0.post0
pytz 2024.1
PyYAML 6.0.1
regex 2024.5.15
requests 2.32.3
safetensors 0.4.3
setuptools 69.5.1
six 1.16.0
SQLAlchemy 2.0.30
sympy 1.12.1
tokenizers 0.19.1
torch 2.3.0
torchmetrics 1.4.0.post0
tqdm 4.66.4
transformers 4.41.1
triton 2.3.0
typing_extensions 4.12.0
tzdata 2024.1
urllib3 2.2.1
wheel 0.43.0
The problem is that you have both onnxruntime-genai as well as onnxruntime-genai-cuda installed.
Both python packages will install the necessary dlls inside theonnxruntime_genai folder in your python site-packages. As a result, depending on the order of installation, you will only have one of the two packages installed.
In your case, it looks like the cpu package dlls has overridden the cuda package dlls causing the error you're seeing. I would recommend uninstalling both of them and only re-installing one of them.
@baijumeswani when I run pip uninstall onnxruntime-genai, and run the code again, I got this message:
(onnx) (base) iwaitu@DESKTOP-RDS3VMA:~/work/onnx$ /home/iwaitu/anaconda3/envs/onnx/bin/python /home/iwaitu/work/onnx/demo_cuda.py
Traceback (most recent call last):
File "/home/iwaitu/work/onnx/demo_cuda.py", line 3, in <module>
model = og.Model('cuda/cuda-int4-rtn-block-32')
AttributeError: module 'onnxruntime_genai' has no attribute 'Model'
so I reinstalled onnxruntime-genai-cuda and run the code , I got this message:
Traceback (most recent call last):
File "/home/iwaitu/anaconda3/envs/onnx/lib/python3.10/site-packages/onnxruntime_genai/__init__.py", line 11, in <module>
from onnxruntime_genai.onnxruntime_genai import *
ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/iwaitu/work/onnx/demo_cuda.py", line 1, in <module>
import onnxruntime_genai as og
File "/home/iwaitu/anaconda3/envs/onnx/lib/python3.10/site-packages/onnxruntime_genai/__init__.py", line 14, in <module>
from onnxruntime_genai.onnxruntime_genai import *
ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory
this is my nvcc version
(onnx) iwaitu@DESKTOP-RDS3VMA:~/work/onnx$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:18:24_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0