Release for ArrayFire 3.9 ?
mboisson opened this issue · comments
Is there a release of Arrayfire-python which is compatible with ArrayFire 3.9 ?
We have some big changes incoming for python 3.9 and should have new wheels uploaded within a few weeks!
Among the changes should be the addition of the data-apis array api atop of ArrayFire. We're also trying to modernize the library to current python standards with proper type hints, formatting, testing and all the other python goodies. The binaries will also be moved to a separate repo so we can de-couple the life of the python library from the binary releases of the main c++ repo. The new oneapi backend will also be exposed.
This is an ambitious change for us so community input is greatly appreciated before things break.
Things have been changing a lot so we've been keeping them out of the main project for now until we've verified the new approach will work. I'll have WIP repos for you within a few days and then we'll open a PR for community review a bit after that. Stay tuned!
Will arrayfire-python version 3.8 work with arrayfire 3.9 ?
Will arrayfire-python version 3.8 work with arrayfire 3.9 ?
The new version of the arrayfire-python will have a new API by default but will also support legacy API for some time to give users some space to adopt the new changes. So, the short answer is Yes, but minor interface changes will be required. E.g., import arrayfire as af will be changed for import arrayfire.lefacy_api as af to avoid import issues. Please, note, that this example is subject to change. New features are on the way :)
I meant the other way around. I am currently building arrayfire 3.9. Can I install the arrayfire-python wrappers version 3.8 on top of it ?
The current wrapper can pick up the new 3.9 version and all existing functionality should work. You may need to specify AF_PATH to your build to get it to pick up your binaries. Any new features in 3.9, such as asynchronous reduction functions, won't be exposed with the current wrapper.
So, I installed arrayfire 3.9.0, then I installed arrayfire-python 3.8 on top of it, but it won't import:
$ AF_VERBOSE_LOADS=1 python -c "import arrayfire" |& grep OSError
OSError: /home/mboisson/.local/easybuild/software/2020/avx2/CUDA/gcc9/cuda12.2/arrayfire/3.9.0/lib64/libforge.so.1: cannot open shared object file: No such file or directory
OSError: /home/mboisson/.local/easybuild/software/2020/avx2/CUDA/gcc9/cuda12.2/arrayfire/3.9.0/lib/python3.8/site-packages/arrayfire-3.8.0-py3.8.egg/arrayfire/libforge.so.1: cannot open shared object file: No such file or directory
OSError: /cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.10/lib64/libforge.so.1: cannot open shared object file: No such file or directory
OSError: libforge.so.1: cannot open shared object file: No such file or directory
OSError: libspdlog.so.1: cannot open shared object file: No such file or directory
OSError: /home/mboisson/.local/easybuild/software/2020/avx2/CUDA/gcc9/cuda12.2/arrayfire/3.9.0/lib/python3.8/site-packages/arrayfire-3.8.0-py3.8.egg/arrayfire/libafcpu.so.3: cannot open shared object file: No such file or directory
OSError: /cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.10/lib64/libafcpu.so.3: cannot open shared object file: No such file or directory
OSError: libafcpu.so.3: cannot open shared object file: No such file or directory
OSError: libspdlog.so.1: cannot open shared object file: No such file or directory
OSError: /home/mboisson/.local/easybuild/software/2020/avx2/CUDA/gcc9/cuda12.2/arrayfire/3.9.0/lib/python3.8/site-packages/arrayfire-3.8.0-py3.8.egg/arrayfire/libafopencl.so.3: cannot open shared object file: No such file or directory
OSError: /cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.10/lib64/libafopencl.so.3: cannot open shared object file: No such file or directory
OSError: libafopencl.so.3: cannot open shared object file: No such file or directory
OSError: libspdlog.so.1: cannot open shared object file: No such file or directory
OSError: /home/mboisson/.local/easybuild/software/2020/avx2/CUDA/gcc9/cuda12.2/arrayfire/3.9.0/lib/python3.8/site-packages/arrayfire-3.8.0-py3.8.egg/arrayfire/libafcuda.so.3: cannot open shared object file: No such file or directory
OSError: /cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.10/lib64/libafcuda.so.3: cannot open shared object file: No such file or directory
OSError: libafcuda.so.3: cannot open shared object file: No such file or directory
OSError: libspdlog.so.1: cannot open shared object file: No such file or directory
OSError: /home/mboisson/.local/easybuild/software/2020/avx2/CUDA/gcc9/cuda12.2/arrayfire/3.9.0/lib/python3.8/site-packages/arrayfire-3.8.0-py3.8.egg/arrayfire/libaf.so.3: cannot open shared object file: No such file or directory
OSError: /cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.8.10/lib64/libaf.so.3: cannot open shared object file: No such file or directory
OSError: libaf.so.3: cannot open shared object file: No such file or directory
The variable AF_PATH is defined as such, and contains these shared objects:
$ echo $AF_PATH
/home/mboisson/.local/easybuild/software/2020/avx2/CUDA/gcc9/cuda12.2/arrayfire/3.9.0
$ ls $AF_PATH/lib64
libafcpu.debug libafcpu.so libafcpu.so.3 libafcpu.so.3.9.0 libafcuda.debug libafcuda.so libafcuda.so.3 libafcuda.so.3.9.0 libaf.debug libafopencl.debug libafopencl.so libafopencl.so.3 libafopencl.so.3.9.0 libaf.so libaf.so.3 libaf.so.3.9.0
It appears that it is not looking into $AF_PATH/lib64 to find libafcpu.so.3, libafopencl.so.3, libaf.so.3, libafcuda.so.3 (those libraries are there), and it appears that the install forgot to package libspdlog.so.1 and libforge.so.1 when installing.
Any idea on how to fix this ?
I've tested the current wrapper with the latest 3.9 release and was able to get it to work without any effort other than setting AF_PATH to the installation directory. It looks like you're missing a few of libraries in that are included with the release. You can run ldd whichever.so to see what libraries aren't linked. Forge, spdlog, and a few more shouldn't be strictly necessary for the wrapper to function. Just for reference, here is a list of what is included with the official binaries:
libafcpu.debug libaf.debug libaf.so libcudnn_cnn_train.so.8.9.4 libcusparse.so.12 libmkl_avx2.so.2 libmkl_intel_lp64.so.2 libnvJitLink.so.12 libsvml.so
libafcpu.so libafoneapi.so libaf.so.3 libcudnn_ops_infer.so.8.9.4 libcusparse.so.12.1.2.129 libmkl_avx512.so.2 libmkl_mc3.so.2 libnvJitLink.so.12.2.128 libsycl.so.6
libafcpu.so.3 libafoneapi.so.3 libaf.so.3.9.0 libcudnn_ops_train.so.8.9.4 libforge.so libmkl_avx.so.2 libmkl_mc.so.2 libnvrtc-builtins.so.12.2 libsycl.so.6.2.0
libafcpu.so.3.9.0 libafoneapi.so.3.9.0 libcublasLt.so.12 libcudnn.so.8.9.4 libforge.so.1 libmkl_core.so libmkl_rt.so libnvrtc.so.12 libtbb.so
libafcuda.debug libafopencl.debug libcublasLt.so.12.2.4.5 libcufft.so.11 libforge.so.1.0.8 libmkl_core.so.2 libmkl_rt.so.2 libnvrtc.so.12.2.128 libtbb.so.12
libafcuda.so libafopencl.so libcublas.so.12 libcufft.so.11.0.8.91 libimf.so libmkl_def.so.2 libmkl_sycl.so.3 libOpenCL.so.1 libtbb.so.12.10
libafcuda.so.3 libafopencl.so.3 libcublas.so.12.2.4.5 libcusolver.so.11 libintlc.so.5 libmkl_intel_ilp64.so.2 libmkl_tbb_thread.so libOpenCL.so.1.0.0 libtbb.so.2
libafcuda.so.3.9.0 libafopencl.so.3.9.0 libcudnn_cnn_infer.so.8.9.4 libcusolver.so.11.5.1.129 libirng.so libmkl_intel_lp64.so libmkl_tbb_thread.so.2 libpi_opencl.so
If you are building from scratch, make sure to build with the AF_INSTALL_STANDALONE=ON cmake option before running make install so all additional dependencies are added to your install directory. Are your current paths pointing to the install directory or the build directory? Some of the OSError for the files that do exist seem a bit strange, maybe there's some permissions issue?
I think all of the errors are related to libspdlog.so.1 and libforge.so.1 being missing from the install directory. The AF libraries are linking on those, which are presumably built as part of the build process, but those are not installed, therefore the AF libraries can't be loaded.
I will try with AF_INSTALL_STANDALONE=ON to see if it installs those. Hopefully it does not also package the CUDA libraries like in your list, those should not be packaged with it, as they are installed on the system.
AF_INSTALL_STANDALONE=ON did not fix it. These are the libraries in the lib64 folder:
[mboisson@build-node ~]$ ls /home/mboisson/.local/easybuild/software/2020/avx2/CUDA/gcc9/cuda12.2/arrayfire/3.9.0/lib64
libafcpu.debug libafcpu.so.3.9.0 libafcuda.so.3 libafopencl.debug libafopencl.so.3.9.0 libaf.so.3.9.0 libcudnn_cnn_infer.so.8.9.5 libcudnn_ops_train.so.8.9.5 libcusolver.so.12 libnvrtc.so.12.2
libafcpu.so libafcuda.debug libafcuda.so.3.9.0 libafopencl.so libaf.so libcublasLt.so.12 libcudnn_cnn_train.so.8.9.5 libcudnn.so.8.9.5 libcusparse.so.12 libOpenCL.so.1
libafcpu.so.3 libafcuda.so libaf.debug libafopencl.so.3 libaf.so.3 libcublas.so.12 libcudnn_ops_infer.so.8.9.5 libcufft.so.12 libnvrtc-builtins.so.12.2
it installed libOpenCL.so.1, libcublas.so.12 (which it should not have, so that option is actually broken for us) however, but not libforge.so.1 or libspdlog.so.1
You will need to temporarily set AF_WITH_FMT_HEADER_ONLY=ON , AF_WITH_SPDLOG_HEADER_ONLY=ON to avoid those dependencies. The non-header versions of those libraries have been updated upstream recently and do need to be fixed in the main ArrayFire repo. We should be addressing those shortly.
Forge is not required for the wrapper to run and can also be disabled with the AF_BUILD_FORGE=OFF option. The additional cuda packages are installed on your dev system, however, ArrayFire's only requirement is for the user to have a valid driver rather than install the whole cuda toolkit which is why those are included in the standalone install. Closing issue for now as all issues are upstream and not with wrapper. Feel free to reopen if you run into anything else.