The llama-cpp-python installed using the following method cannot find llama.dll, but the actual llama.dll exists.

Question

The llama-cpp-python installed using the following method cannot find llama.dll, but the actual llama.dll exists.

chengjia604 opened this issue 7 months ago · comments

installation method：
https://github.com/lperezmo/clblast-llama-cpp-python
Download llama_cpp_python：

error:
Traceback (most recent call last):
File "d:\python\Lib\site-packages\llama_cpp\llama_cpp.py", line 72, in _load_shared_library
return ctypes.CDLL(str(lib_path), **cdll_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\python\Lib\ctypes_init.py", line 376, in init
self._handle = _dlopen(self._name, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: Could not find module 'd:\python\Lib\site-packages\llama_cpp\llama.dll' (or one of its dependencies). Try using the full path with constructor syntax.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "d:\python\Lib\site-packages\llama_cpp\llama_cpp.py", line 86, in
_lib = _load_shared_library(_lib_base_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "d:\python\Lib\site-packages\llama_cpp\llama_cpp.py", line 75, in _load_shared_library
raise RuntimeError(f"Failed to load shared library '{_lib_path}': {e}")
RuntimeError: Failed to load shared library 'd:\python\Lib\site-packages\llama_cpp\llama.dll': Could not find module 'd:\python\Lib\site-packages\llama_cpp\llama.dll' (or one of its dependencies). Try using the full path with constructor syntax.

jreves · Answer 1 · Fri Dec 08 2023 01:32:06 GMT+0800 (China Standard Time)

I have the same issue, after finally getting a clean install with HIPBLAS. My standalone install of llama_cpp works, and runs on the GPU. Trying to import the library in jupyter - using this kernel - produces the same error as above. I also have llama.dll and llama.lib in my site-packages\llama_cpp directory.

Just running python interactively in my virtual environment, I am able to import llama without error.

jreves · Answer 2 · Mon Dec 11 2023 00:13:36 GMT+0800 (China Standard Time)

I discovered something that might be useful and relevant for this issue: This error suggests that the dll can't be found, but it's obviously there in the correct location. The error is also generated if a dependency can't be found - and that was the case for me. I'm not sure how you compiled, but in my case I referenced the HIP code to access my AMD GPU. Turns out - I had not updated my windows system PATH to include the path to the directory for the dependent BLAS libraries. Adding those directories to my system path resolved this issue for me. Worth checking!

Andrei · Answer 3 · Fri Dec 22 2023 03:28:26 GMT+0800 (China Standard Time)

@jreves that's correct, it's likely a dependency. If you check out llama_cpp.py line 58 actually adds the CUDA_PATH as a dll search directory, not sure if AMD has a similar common path variable but would be happy to add it if it helps.

jreves · Answer 4 · Sat Dec 23 2023 04:15:06 GMT+0800 (China Standard Time)

@abetlen , yes there's a similar path for HIP installs on native Windows. The HIP directory is available as %HIP_PATH%, and the libraries are in %HIP_PATH$lib. The utilities - including the clang and clang++ compilers are in %HIP_PATH%bin.

%HIP_PATH% references the specific version as a part of the directory structure, so on my system - for example - %HIP_PATH$ expands to C:\Program Files\AMD\ROCm\5.5. The hipblas.dll file is in the bin directory.

There's a useful utility in the bin directory, as well: hipinfo.exe. Here's an example of the output:
c:\Program Files\AMD\ROCm\5.5\bin>hipinfo

device# 0
Name: AMD Radeon RX 6900 XT
pciBusID: 47
pciDeviceID: 0
pciDomainID: 0
multiProcessorCount: 40
maxThreadsPerMultiProcessor: 2048
isMultiGpuBoard: 0
clockRate: 2105 Mhz
memoryClockRate: 1000 Mhz
memoryBusWidth: 0
totalGlobalMem: 15.98 GB
totalConstMem: 2147483647
sharedMemPerBlock: 64.00 KB
canMapHostMemory: 1
regsPerBlock: 0
warpSize: 32
l2CacheSize: 4194304
computeMode: 0
maxThreadsPerBlock: 1024
maxThreadsDim.x: 1024
maxThreadsDim.y: 1024
maxThreadsDim.z: 1024
maxGridSize.x: 2147483647
maxGridSize.y: 65536
maxGridSize.z: 65536
major: 10
minor: 3
concurrentKernels: 1
cooperativeLaunch: 0
cooperativeMultiDeviceLaunch: 0
isIntegrated: 0
maxTexture1D: 16384
maxTexture2D.width: 16384
maxTexture2D.height: 16384
maxTexture3D.width: 2048
maxTexture3D.height: 2048
maxTexture3D.depth: 2048
isLargeBar: 0
asicRevision: 0
maxSharedMemoryPerMultiProcessor: 64.00 KB
clockInstructionRate: 1000.00 Mhz
arch.hasGlobalInt32Atomics: 1
arch.hasGlobalFloatAtomicExch: 1
arch.hasSharedInt32Atomics: 1
arch.hasSharedFloatAtomicExch: 1
arch.hasFloatAtomicAdd: 1
arch.hasGlobalInt64Atomics: 1
arch.hasSharedInt64Atomics: 1
arch.hasDoubles: 1
arch.hasWarpVote: 1
arch.hasWarpBallot: 1
arch.hasWarpShuffle: 1
arch.hasFunnelShift: 0
arch.hasThreadFenceSystem: 1
arch.hasSyncThreadsExt: 0
arch.hasSurfaceFuncs: 0
arch.has3dGrid: 1
arch.hasDynamicParallelism: 0
gcnArchName: gfx1030
peers:
non-peers: device#0

memInfo.total: 15.98 GB
memInfo.free: 15.86 GB (99%)

Andrei · Answer 5 · Sat Dec 23 2023 04:32:06 GMT+0800 (China Standard Time)

@jreves perfect, I've added HIP_PATH to the dll search directories for windows users if it's defined in their environment. Should be in the next release

Andrei · Answer 6 · Fri Jan 12 2024 05:51:46 GMT+0800 (China Standard Time)

@chengjia604 does that resolve your issue as well?

QQCif · Answer 7 · Tue Feb 20 2024 16:42:05 GMT+0800 (China Standard Time)

I tried to use the prebuilt binary provided by llama.cpp repo. After setting LLAMA_CPP_LIB to the path of llama.dll I am getting the same error. cublas64_12.dll, cublasLt64_12, cudart64_12.dll are all present in the same folder as llama.dll. However if I tried to add llama.cpp binary folder to PATH, then it works.