Using onnxruntime-genai-directml without CUDA

[Matthieu Maitre]

I pip installed the DirectML version of onnxruntime-genai (onnxruntime-genai-directml==0.3.0rc2) hoping to avoid having to install CUDA but importing the library fails mentioning there is no CUDA. Is there a way to use ONNX with DirectML and not CUDA?

Traceback (most recent call last):
  File "<snip>\envs\phi3\Lib\site-packages\onnxruntime_genai\__init__.py", line 11, in <module>
    from onnxruntime_genai.onnxruntime_genai import *
ImportError: DLL load failed while importing onnxruntime_genai: A dynamic link library (DLL) initialization routine failed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<snip>\Phi3\phi3-qa.py", line 1, in <module>
    import onnxruntime_genai as og
  File "<snip>\envs\phi3\Lib\site-packages\onnxruntime_genai\__init__.py", line 13, in <module>
    _dll_directory.add_dll_directory()
  File "<snip>\envs\phi3\Lib\site-packages\onnxruntime_genai\_dll_directory.py", line 21, in add_dll_directory
    raise ImportError("Could not find the CUDA libraries. Please set the CUDA_PATH environment variable.")
ImportError: Could not find the CUDA libraries. Please set the CUDA_PATH environment variable.

[Baiju Meswani]

Although the error reports that the CUDA library is not found, the real problem is that one of the dependency DLL was not found:

ImportError: DLL load failed while importing onnxruntime_genai: A dynamic link library (DLL) initialization routine failed.

In the event of a failed onnxruntime_genai import (with the missing DLL error), ort-genai assumes that the installed package is the CUDA one and the missing DLL error is from Python not being able to find the CUDA DLLs. In such a scenario, ORT GenAI tries to add a search path for the missing CUDA DLL before erroring out if still not found.

Unfortunately, the error is misleading when the missing DLL is some other DLL (which seems to be the case for you). Are you able to identify the missing DLL? Could you share the output of this command?

dumpbin /dependents <snip>\envs\phi3\Lib\site-packages\onnxruntime_genai\onnxruntime_genai.cp311-win_amd64.pyd

Please replace the cp311 in the command with the version of python you're using.

[Matthieu Maitre]

Here is the output of dumpbin. I tried Dependencies on the DLL (Dependency Walker just hanged) and all the dependencies seem to resolve. Maybe an interop issue with Python 12?

Dump of file <snip>\envs\phi3\lib\site-packages\onnxruntime_genai\onnxruntime_genai.cp312-win_amd64.pyd

File Type: DLL

  Image has the following dependencies:

    onnxruntime.dll
    python312.dll
    d3d12.dll
    dxgi.dll
    KERNEL32.dll
    MSVCP140.dll
    CONCRT140.dll
    VCRUNTIME140_1.dll
    VCRUNTIME140.dll
    api-ms-win-crt-heap-l1-1-0.dll
    api-ms-win-crt-string-l1-1-0.dll
    api-ms-win-crt-runtime-l1-1-0.dll
    api-ms-win-crt-stdio-l1-1-0.dll
    api-ms-win-crt-filesystem-l1-1-0.dll
    api-ms-win-crt-convert-l1-1-0.dll
    api-ms-win-crt-math-l1-1-0.dll
    api-ms-win-crt-locale-l1-1-0.dll
    api-ms-win-crt-environment-l1-1-0.dll

[Baiju Meswani]

We've heard some others have also faced this issue. Although I haven't been able to reproduce it. Some of those who were able to reproduce this mentioned that the missing lib was zlib.dll.

I will find a machine where I can reproduce this and fix it. Thanks for sharing the details.

[Yufeng Li]

@mmaitre314, it turns out to be that conda uses old vc runtime and it is not compatible with new visual studio compiler. Could you please try run "conda install conda-forge::vs2015_runtime" to upgrade your vc runtime version.

We are working on a hot fix on it too.

[Baiju Meswani]

Some more context:

MSVC team recently updated their stl library and we compile our source code in onnxruntime-genai against the updated stl library. But the conda env vc runtime dlls are older than the expected version. This causes the problem A dynamic link library (DLL) initialization routine failed.

What can be done to address this?

In due course, conda dlls will be updated. To address this on our end, we added the PR #599 so as to extend the support to a slightly older vc runtime dll version. But this won't help if the vc runtime dll is even older. This will be a part of the 0.3.0 ort-genai release.

An alternative is to use the command @yufenglee shared and update the vc runtime version conda install conda-forge::vs2015_runtime. This should resolve the issue.

Please give it a try and let us know how that goes. I'll close this issue now. Do let us know if you encounter more issues.

[Matthieu Maitre]

Confirming installing vs2015_runtime resolved the issue.