Windows Arrayfire C library loading error even after using ctypes to help debug
hyliu1989 opened this issue · comments
It seems there is a missing DLL problem that prevents Arrayfire from running on a new installed Windows machine.
The Python (3.7.1) is installed through Anaconda, and the Arrayfire binary installation file and Visual C++ Runtime 2015 are also installed. All in x64 version (so not exactly the same as #166).
However, when importing in Python by import arrayfire, I still get the following error
RuntimeError: Could not load any ArrayFire libraries.
Please look at https://github.com/arrayfire/arrayfire-python/wiki for more information.
To distill the cause of the error, I use ctypes to import the arrayfire DLL to see if it is successful, only to find that the DLL file cannot be loaded either
In [2]: import ctypes
In [3]: import os
In [4]: dll_path = r'C:\Program Files\ArrayFire\v3\lib\afcpu.dll'
In [5]: os.path.exists(dll_path)
Out[5]: True
In [6]: ctypes.cdll.LoadLibrary(dll_path)
---------------------------------------------------------------------------
OSError Traceback (most recent call last)
<ipython-input-6-97edb3acce44> in <module>
----> 1 ctypes.cdll.LoadLibrary(dll_path)
C:\ProgramData\Anaconda3\lib\ctypes\__init__.py in LoadLibrary(self, name)
432
433 def LoadLibrary(self, name):
--> 434 return self._dlltype(name)
435
436 cdll = LibraryLoader(CDLL)
C:\ProgramData\Anaconda3\lib\ctypes\__init__.py in __init__(self, name, mode, handle, use_errno, use_last_error)
354
355 if handle is None:
--> 356 self._handle = _dlopen(self._name, mode)
357 else:
358 self._handle = handle
OSError: [WinError 126] The specified module could not be found
It seems that a dependent DLL is missing but the error message is not very helpful....
Missing DLL found and solved by another working machine.
Thanks to this site, which introduces Sysinternals Process Monitor, that I am able to use to find out which DLL is missing.
It turns out the missing one is ucrtbased.dll which should be located in system32 after installing one of the update versions of Visual Studio 2015. I am able to run Arrayfire on the newly set up computer after copying that DLL from another PC that has a working Arrayfire.
There is one with a similar name ucrtbase.dll in system32, but I am not sure what's the difference in them. I also did not figure out which update version of VS2015 to install in order to have ucrtbased.dll...
@umar456 doesnt arrayfire libs have documentation about installing visual studio runtime ?
It appears that the missing DLL ucrtbased.dll is the debug model library of ucrtbase.dll which will be installed by selecting "Common Tools for Visual C++ 2015" during VC2015 installation. (https://stackoverflow.com/questions/33743493/why-visual-studio-2015-cant-run-exe-file-ucrtbased-dll)
Maybe we can update the instruction page with this tiny selection or make the arrayfire C library independent of the debug mode DLL.
@hyliu1989 where did you get arrayfire libs from ?
I got it from the binary provided in https://arrayfire.com/download/. At the time of download, it was Windows v3.6.2.
Same problem here.
@hyliu1989 Could you please provide a download link for the missing dll or the file itself? That would be great, thank you...
@ZacDiggum quote from my colleague:
Looking back, when we installed VS2015 and selected what components to install, we should have chosen to install the C language, and there is a “common tools” checkbox under C that we should have selected. If we checked that one, the missing dll would be installed. I tried it and it worked.
@pavanky
Is this ucrtbased.dll really necessary? I can't remember to have to install VS 2015 in order to get older versions of Arrayfire to work. Maybe you could drop the dependency or deploy the DLL with the AF installer?
@umar456 can you help figure this out?
Sorry guys. I can look into this early next week.
I have been looking into the transitive dependencies for afcpu.dll & af.dll and both of them list only ucrtbase.dll , I do not see ucrtbased.dll anywhere in the entire dependency list. I used dependency walker to do this. The debug dll has d suffix and it is unlikely that a dll looking for ucrtbase.dll will pickup ucrtbased.dll.
Can you please do the same for the afcpu.dll on your installation and see which ucrtbase dll is it picking up. You can download(portable exe) from here http://www.dependencywalker.com/ Once the dll is loaded into the dependency walker(this can time few seconds to minutes), save the output to a text file. You can search that txt file normally to see if ucrtbased.dll is listed in any of the indirect dependencies.
@hyliu1989 @ZacDiggum Also, can you please tell us how did you install arrayfire-python package ? via conda or pip ?
I have two different setups at home and at work:
at home: Win 10 Pro 64, AF 3.6.2 -> error message like @hyliu1989
at work: Win 7 Pro 64, AF 3.5.1 -> 'Intel MKL FATAL ERROR: Cannot load mkl_intel_thread.dll.' (this is a new issue)
I installed af-python via pip (v3.6.20181017) into identical Python 3.6.5 distros (Winpython) in both cases.
I'll dig into it and try dependencywalker today.
@ZacDiggum Regarding the mkl loading issue, is the path to dll files of ArrayFire installation added to PATH environment variable ? Since, mkl does runtime loading of it's own dlls, it wouldn't work if %AF_PATH%/lib is not added to PATH.
Yes, the ...af/v3/lib folder is in PATH. Somehow this MKL error is new.
for dependencywalker:
af.dll seems to be functional
afcpu.dll says:
this is for the Win7 Pro 64 system
afaik, you can't really search the list of dlls directly in the dependency-walker GUI. Hence I asked you to save that as text file - save icon in the GUI. The text file can be searched for names like ucrtbased.dll or ucrtbase.dll
Sorry, here is the output from dependencywalker regarding afcpu.dll on the win 7 system with the mkl error:
@9prady9 I also installed arrayfire-python through pip. Also in my previous thread, I also tried ctypes to bypass python wrapper to see if the problem is in C library.
Here are the two files from dependency walker
af.txt
afcpu.txt.zip
Also I ran the process monitor and recorded what happened after import ctypes; ctypes.cdll.LoadLibrary(r'c:\Program Files\ArrayFire\v3\lib\af.dll') and import ctypes; ctypes.cdll.LoadLibrary(r'c:\Program Files\ArrayFire\v3\lib\afcpu.dll')
af_procmonitor.txt
afcpu_procmonitor.txt
From here you can clearly see that the python process is trying to load ucrtbased.dll and the requirement to load ucrtbased.dll might buried in some other dll that afcpu.dll depends.
Thank you for the traces and logs, I found the dll that is loading debug VC runtime dll which is in turn loading the ucrtbase debug dll. It is an upstream dependency (glbindingd.dll) but not afcpu.dll by itself. It is interesting though why on the windows machine I tested, I didn't find ucrtbased.dll for afcpu.dll dependency walk. In any case, I think I know the reason where this debug dll is coming from. I will fix it and update the installers for 3.6.2 as soon as we can.
Having said that, please do note that this is no longer an issue on master and will not be an issue in future releases.
Note: I found the reason why my installed version didn't look for debug version of ucrtbase dll, because it was a no-gl ArrayFire installer binary.
@ZacDiggum @hyliu1989 There is a work around for your current situation if you don't use graphics functionality from ArrayFire. Download the http://arrayfire.s3.amazonaws.com/3.6.2/ArrayFire-no-gl-v3.6.2.exe and use that, that won't look for debug version of dlls.
@umar456 We should probably just do a new fix release instead of regenerating now and then again doing it for 3.6.3 again.
Nvm, we should still fix 3.6.2 gl installer if someone wants to stick to it.