Python binding installation fails
regevs opened this issue · comments
I am trying to install zfp with Python bindings. I have a conda environment snakemake which I want to use. I run cmake to get:
$ cmake -DBUILD_ZFPY=ON -DPYTHON_LIBRARY=/home/rs2145/rs2145/software/anaconda3/envs/snakemake/lib/libpython3.9.so -DPYTHON_INCLUDE_DIR=/home/rs2145/rs2145/software/anaconda3/envs/snakemake/include/python3.9/ ..
-- The C compiler identification is GNU 8.5.0
-- The CXX compiler identification is GNU 8.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Compiling with C standard: 99
-- Compiling with C++ standard: 98
-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5") found components: C
-- Performing Test HAVE_MATH
-- Performing Test HAVE_MATH - Failed
-- Performing Test HAVE_LIBM_MATH
-- Performing Test HAVE_LIBM_MATH - Success
-- Found PythonLibs: /home/rs2145/rs2145/software/anaconda3/envs/snakemake/lib/libpython3.9.so (found version "3.9.9")
-- Found PythonInterp: /home/rs2145/rs2145/software/anaconda3/bin/python3.9 (found version "3.9.13")
-- Found NumPy: /usr/include (found version "1.21.5")
-- Found Cython: /home/rs2145/rs2145/software/anaconda3/bin/cython (Required is at least version "0.28")
-- Performing Test Weak Link MODULE -> SHARED (gnu_ld_ignore) - Success
-- Configuring done
-- Generating done
-- Build files have been written to: /home/rs2145/rs2145/git/zfp/build
Then make to get:
$ make
[ 2%] Building C object src/CMakeFiles/zfp.dir/zfp.c.o
[ 4%] Building C object src/CMakeFiles/zfp.dir/bitstream.c.o
[ 6%] Building C object src/CMakeFiles/zfp.dir/encode1f.c.o
[ 9%] Building C object src/CMakeFiles/zfp.dir/encode1d.c.o
[ 11%] Building C object src/CMakeFiles/zfp.dir/encode1i.c.o
[ 13%] Building C object src/CMakeFiles/zfp.dir/encode1l.c.o
[ 15%] Building C object src/CMakeFiles/zfp.dir/decode1f.c.o
[ 18%] Building C object src/CMakeFiles/zfp.dir/decode1d.c.o
[ 20%] Building C object src/CMakeFiles/zfp.dir/decode1i.c.o
[ 22%] Building C object src/CMakeFiles/zfp.dir/decode1l.c.o
[ 25%] Building C object src/CMakeFiles/zfp.dir/encode2f.c.o
[ 27%] Building C object src/CMakeFiles/zfp.dir/encode2d.c.o
[ 29%] Building C object src/CMakeFiles/zfp.dir/encode2i.c.o
[ 31%] Building C object src/CMakeFiles/zfp.dir/encode2l.c.o
[ 34%] Building C object src/CMakeFiles/zfp.dir/decode2f.c.o
[ 36%] Building C object src/CMakeFiles/zfp.dir/decode2d.c.o
[ 38%] Building C object src/CMakeFiles/zfp.dir/decode2i.c.o
[ 40%] Building C object src/CMakeFiles/zfp.dir/decode2l.c.o
[ 43%] Building C object src/CMakeFiles/zfp.dir/encode3f.c.o
[ 45%] Building C object src/CMakeFiles/zfp.dir/encode3d.c.o
[ 47%] Building C object src/CMakeFiles/zfp.dir/encode3i.c.o
[ 50%] Building C object src/CMakeFiles/zfp.dir/encode3l.c.o
[ 52%] Building C object src/CMakeFiles/zfp.dir/decode3f.c.o
[ 54%] Building C object src/CMakeFiles/zfp.dir/decode3d.c.o
[ 56%] Building C object src/CMakeFiles/zfp.dir/decode3i.c.o
[ 59%] Building C object src/CMakeFiles/zfp.dir/decode3l.c.o
[ 61%] Building C object src/CMakeFiles/zfp.dir/encode4f.c.o
[ 63%] Building C object src/CMakeFiles/zfp.dir/encode4d.c.o
[ 65%] Building C object src/CMakeFiles/zfp.dir/encode4i.c.o
[ 68%] Building C object src/CMakeFiles/zfp.dir/encode4l.c.o
[ 70%] Building C object src/CMakeFiles/zfp.dir/decode4f.c.o
[ 72%] Building C object src/CMakeFiles/zfp.dir/decode4d.c.o
[ 75%] Building C object src/CMakeFiles/zfp.dir/decode4i.c.o
[ 77%] Building C object src/CMakeFiles/zfp.dir/decode4l.c.o
[ 79%] Linking C shared library ../lib64/libzfp.so
[ 79%] Built target zfp
[ 81%] Generating C source python/zfpy.c
Scanning dependencies of target zfpy
[ 84%] Building C object python/CMakeFiles/zfpy.dir/zfpy.c.o
In file included from /usr/include/numpy/ndarraytypes.h:1816,
from /usr/include/numpy/ndarrayobject.h:18,
from /usr/include/numpy/arrayobject.h:4,
from /home/rs2145/rs2145/git/zfp/build/python/zfpy.c:751:
/usr/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it by " \
^~~~~~~
[ 86%] Linking C shared module ../bin/zfpy.cpython-39-x86_64-linux-gnu.so
[ 86%] Built target zfpy
[ 88%] Building C object utils/CMakeFiles/zfpcmd.dir/zfp.c.o
[ 90%] Linking C executable ../bin/zfp
[ 90%] Built target zfpcmd
[ 93%] Building CXX object tests/CMakeFiles/testviews.dir/testviews.cpp.o
/home/rs2145/rs2145/git/zfp/tests/testviews.cpp: In function ‘int main(int, char**)’:
/home/rs2145/rs2145/git/zfp/tests/testviews.cpp:240:11: internal compiler error: in pop_local_binding, at cp/name-lookup.c:2047
return 0;
^
Please submit a full bug report,
with preprocessed source if appropriate.
See <https://bugs.rockylinux.org/> for instructions.
Preprocessed source stored into /tmp/ccw8OlBO.out file, please attach this to your bugreport.
make[2]: *** [tests/CMakeFiles/testviews.dir/build.make:76: tests/CMakeFiles/testviews.dir/testviews.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:988: tests/CMakeFiles/testviews.dir/all] Error 2
make: *** [Makefile:146: all] Error 2
It looks for numpy in /usr/include/numpy, which is not what it should be using. The numpy installation is at /home/rs2145/rs2145/software/anaconda3/envs/snakemake/lib/python3.9/site-packages/numpy/core/include, which I am not sure how it should find it.
Help appreciated. Thanks.
This is not an issue I'm knowledgable about, but perhaps try specifying the NumPy include directory via Python3_NumPy_INCLUDE_DIR on the CMake line?
Also, the deprecated NumPy warning is to be expected. It rather looks like the issue is some internal compiler error when building testviews, which does not make use of anything Python. Although not recommended, you can always disable these tests via -DBUILD_TESTING=OFF.
Thanks. Can you say explicitly how to specify the NumPy include directory via Python3_NumPy_INCLUDE_DIR on the CMake line?
Add `-DPython3_NumPy_INCLUDE_DIR= /home/rs2145/rs2145/software/anaconda3/envs/snakemake/lib/python3.9/site-packages/numpy/core/include'.
I get:
$ cmake -DBUILD_ZFPY=ON -DPYTHON_LIBRARY=/home/rs2145/rs2145/software/anaconda3/envs/snakemake/lib/libpython3.9.so -DPYTHON_INCLUDE_DIR=/home/rs2145/rs2145/software/anaconda3/envs/snakemake/include/python3.9/ -DPython3_NumPy_INCLUDE_DIR=/home/rs2145/rs2145/software/anaconda3/envs/snakemake/lib/python3.9/site-packages/numpy/core/include ..
-- The C compiler identification is GNU 11.2.0
-- The CXX compiler identification is GNU 11.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/rs2145/rs2145/software/anaconda3/envs/snakemake/bin/x86_64-conda-linux-gnu-cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/rs2145/rs2145/software/anaconda3/envs/snakemake/bin/x86_64-conda-linux-gnu-c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Compiling with C standard: 99
-- Compiling with C++ standard: 98
-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5") found components: C
-- Performing Test HAVE_MATH
-- Performing Test HAVE_MATH - Failed
-- Performing Test HAVE_LIBM_MATH
-- Performing Test HAVE_LIBM_MATH - Success
-- Found PythonLibs: /home/rs2145/rs2145/software/anaconda3/envs/snakemake/lib/libpython3.9.so (found version "3.9.9")
-- Found PythonInterp: /home/rs2145/rs2145/software/anaconda3/envs/snakemake/bin/python3.9 (found version "3.9.9")
-- Found NumPy: /usr/include (found version "1.21.6")
-- Found Cython: /home/rs2145/rs2145/software/anaconda3/envs/snakemake/bin/cython (Required is at least version "0.28")
-- Performing Test Weak Link MODULE -> SHARED (gnu_ld_ignore) - Success
-- Configuring done
-- Generating done
CMake Warning:
Manually-specified variables were not used by the project:
Python3_NumPy_INCLUDE_DIR
-- Build files have been written to: /home/rs2145/rs2145/git/zfp/build
and then:
$ make
[ 2%] Building C object src/CMakeFiles/zfp.dir/zfp.c.o
[ 4%] Building C object src/CMakeFiles/zfp.dir/bitstream.c.o
[ 6%] Building C object src/CMakeFiles/zfp.dir/encode1f.c.o
[ 9%] Building C object src/CMakeFiles/zfp.dir/encode1d.c.o
[ 11%] Building C object src/CMakeFiles/zfp.dir/encode1i.c.o
[ 13%] Building C object src/CMakeFiles/zfp.dir/encode1l.c.o
[ 15%] Building C object src/CMakeFiles/zfp.dir/decode1f.c.o
[ 18%] Building C object src/CMakeFiles/zfp.dir/decode1d.c.o
[ 20%] Building C object src/CMakeFiles/zfp.dir/decode1i.c.o
[ 22%] Building C object src/CMakeFiles/zfp.dir/decode1l.c.o
[ 25%] Building C object src/CMakeFiles/zfp.dir/encode2f.c.o
[ 27%] Building C object src/CMakeFiles/zfp.dir/encode2d.c.o
[ 29%] Building C object src/CMakeFiles/zfp.dir/encode2i.c.o
[ 31%] Building C object src/CMakeFiles/zfp.dir/encode2l.c.o
[ 34%] Building C object src/CMakeFiles/zfp.dir/decode2f.c.o
[ 36%] Building C object src/CMakeFiles/zfp.dir/decode2d.c.o
[ 38%] Building C object src/CMakeFiles/zfp.dir/decode2i.c.o
[ 40%] Building C object src/CMakeFiles/zfp.dir/decode2l.c.o
[ 43%] Building C object src/CMakeFiles/zfp.dir/encode3f.c.o
[ 45%] Building C object src/CMakeFiles/zfp.dir/encode3d.c.o
[ 47%] Building C object src/CMakeFiles/zfp.dir/encode3i.c.o
[ 50%] Building C object src/CMakeFiles/zfp.dir/encode3l.c.o
[ 52%] Building C object src/CMakeFiles/zfp.dir/decode3f.c.o
[ 54%] Building C object src/CMakeFiles/zfp.dir/decode3d.c.o
[ 56%] Building C object src/CMakeFiles/zfp.dir/decode3i.c.o
[ 59%] Building C object src/CMakeFiles/zfp.dir/decode3l.c.o
[ 61%] Building C object src/CMakeFiles/zfp.dir/encode4f.c.o
[ 63%] Building C object src/CMakeFiles/zfp.dir/encode4d.c.o
[ 65%] Building C object src/CMakeFiles/zfp.dir/encode4i.c.o
[ 68%] Building C object src/CMakeFiles/zfp.dir/encode4l.c.o
[ 70%] Building C object src/CMakeFiles/zfp.dir/decode4f.c.o
[ 72%] Building C object src/CMakeFiles/zfp.dir/decode4d.c.o
[ 75%] Building C object src/CMakeFiles/zfp.dir/decode4i.c.o
[ 77%] Building C object src/CMakeFiles/zfp.dir/decode4l.c.o
[ 79%] Linking C shared library ../lib64/libzfp.so
[ 79%] Built target zfp
[ 81%] Generating C source python/zfpy.c
Scanning dependencies of target zfpy
[ 84%] Building C object python/CMakeFiles/zfpy.dir/zfpy.c.o
/home/rs2145/rs2145/git/zfp/build/python/zfpy.c:751:10: fatal error: numpy/arrayobject.h: No such file or directory
751 | #include "numpy/arrayobject.h"
| ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [python/CMakeFiles/zfpy.dir/build.make:82: python/CMakeFiles/zfpy.dir/zfpy.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:936: python/CMakeFiles/zfpy.dir/all] Error 2
make: *** [Makefile:146: all] Error 2
I doesn't seem to pick it up.
Before we spend too much time on this, is /usr/include/numpy a symbolic link, and if so what does it point to?
$ ll /usr/include/numpy
lrwxrwxrwx 1 root root 60 Nov 10 2021 /usr/include/numpy -> /usr/lib64/python2.7/site-packages/numpy/core/include/numpy/
OK, that seems like an issue. It's possible that your FindPython.cmake uses slightly different variable names to locate NumPy. Maybe try variants on Python3_NumPy_INCLUDE_DIR with and without the 3, in all upper case (e.g., PYTHON_NUMPY_INCLUDE_DIR), replacing DIR with DIRS, and specifying this path as an environment variable instead of on the CMake line. I'm really not a Python person, so I'm just guessing here.
@GarrettDMorrison @da-wad Do you have any other suggestions?
@regevs @lindstro the cmake parameter you need to set is I believe NumPy_INCLUDE_DIR (so -DNumPy_INCLUDE_DIR=/path/to/numpy/include). If you have curses cmake installed you can see all the possible options by using ccmake instead of cmake and hitting t to toggle on advanced cmake parameters. Can be helpful in tracking down these sorts of things.
OK, this seems to work, although with a warning:
$ make
[ 2%] Building C object src/CMakeFiles/zfp.dir/zfp.c.o
[ 4%] Building C object src/CMakeFiles/zfp.dir/bitstream.c.o
[ 6%] Building C object src/CMakeFiles/zfp.dir/encode1f.c.o
[ 9%] Building C object src/CMakeFiles/zfp.dir/encode1d.c.o
[ 11%] Building C object src/CMakeFiles/zfp.dir/encode1i.c.o
[ 13%] Building C object src/CMakeFiles/zfp.dir/encode1l.c.o
[ 15%] Building C object src/CMakeFiles/zfp.dir/decode1f.c.o
[ 18%] Building C object src/CMakeFiles/zfp.dir/decode1d.c.o
[ 20%] Building C object src/CMakeFiles/zfp.dir/decode1i.c.o
[ 22%] Building C object src/CMakeFiles/zfp.dir/decode1l.c.o
[ 25%] Building C object src/CMakeFiles/zfp.dir/encode2f.c.o
[ 27%] Building C object src/CMakeFiles/zfp.dir/encode2d.c.o
[ 29%] Building C object src/CMakeFiles/zfp.dir/encode2i.c.o
[ 31%] Building C object src/CMakeFiles/zfp.dir/encode2l.c.o
[ 34%] Building C object src/CMakeFiles/zfp.dir/decode2f.c.o
[ 36%] Building C object src/CMakeFiles/zfp.dir/decode2d.c.o
[ 38%] Building C object src/CMakeFiles/zfp.dir/decode2i.c.o
[ 40%] Building C object src/CMakeFiles/zfp.dir/decode2l.c.o
[ 43%] Building C object src/CMakeFiles/zfp.dir/encode3f.c.o
[ 45%] Building C object src/CMakeFiles/zfp.dir/encode3d.c.o
[ 47%] Building C object src/CMakeFiles/zfp.dir/encode3i.c.o
[ 50%] Building C object src/CMakeFiles/zfp.dir/encode3l.c.o
[ 52%] Building C object src/CMakeFiles/zfp.dir/decode3f.c.o
[ 54%] Building C object src/CMakeFiles/zfp.dir/decode3d.c.o
[ 56%] Building C object src/CMakeFiles/zfp.dir/decode3i.c.o
[ 59%] Building C object src/CMakeFiles/zfp.dir/decode3l.c.o
[ 61%] Building C object src/CMakeFiles/zfp.dir/encode4f.c.o
[ 63%] Building C object src/CMakeFiles/zfp.dir/encode4d.c.o
[ 65%] Building C object src/CMakeFiles/zfp.dir/encode4i.c.o
[ 68%] Building C object src/CMakeFiles/zfp.dir/encode4l.c.o
[ 70%] Building C object src/CMakeFiles/zfp.dir/decode4f.c.o
[ 72%] Building C object src/CMakeFiles/zfp.dir/decode4d.c.o
[ 75%] Building C object src/CMakeFiles/zfp.dir/decode4i.c.o
[ 77%] Building C object src/CMakeFiles/zfp.dir/decode4l.c.o
[ 79%] Linking C shared library ../lib64/libzfp.so
[ 79%] Built target zfp
[ 81%] Generating C source python/zfpy.c
Scanning dependencies of target zfpy
[ 84%] Building C object python/CMakeFiles/zfpy.dir/zfpy.c.o
In file included from /home/rs2145/rs2145/software/anaconda3/envs/snakemake/lib/python3.9/site-packages/numpy/core/include/numpy/ndarraytypes.h:1969,
from /home/rs2145/rs2145/software/anaconda3/envs/snakemake/lib/python3.9/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /home/rs2145/rs2145/software/anaconda3/envs/snakemake/lib/python3.9/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from /home/rs2145/rs2145/git/zfp/build/python/zfpy.c:751:
/home/rs2145/rs2145/software/anaconda3/envs/snakemake/lib/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
17 | #warning "Using deprecated NumPy API, disable it with " \
| ^~~~~~~
[ 86%] Linking C shared module ../bin/zfpy.cpython-39-x86_64-linux-gnu.so
[ 86%] Built target zfpy
[ 88%] Building C object utils/CMakeFiles/zfpcmd.dir/zfp.c.o
[ 90%] Linking C executable ../bin/zfp
[ 90%] Built target zfpcmd
[ 93%] Building CXX object tests/CMakeFiles/testviews.dir/testviews.cpp.o
[ 95%] Linking CXX executable ../bin/testviews
[ 95%] Built target testviews
[ 97%] Building CXX object tests/CMakeFiles/testzfp.dir/testzfp.cpp.o
[100%] Linking CXX executable ../bin/testzfp
[100%] Built target testzfp
However I can't import zfpy:
$ python
Python 3.9.9 | packaged by conda-forge | (main, Dec 20 2021, 02:41:03)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import zfpy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'zfpy'
>>>
I propose adding the -DNumPy_INCLUDE_DIR flag to the docs at: https://zfp.readthedocs.io/en/release1.0.0/installation.html#c.BUILD_ZFPY
Thanks!
Did you do a make install also? This may require setting CMAKE_INSTALL_PREFIX to point to where you want the package installed.
Unless you have a need to install zfpy from source, you could also conda install -c conda-forge zfpy.
Installing with conda works. Thanks!
Great to hear. I will go ahead and close this issue. Please reopen if you want to pursue installing from source.
I want to follow up on this thread. I am trying to build the python binding. After cmake install, I obtained the zfpy.cpython-37m-x86_64-linux-gnu.so shared object file.
However, when I tried to load it in python using CDLL, I cannot seem to load the compress_numpy function.
The code looks like:
import ctypes
# Load the shared object library
zfp = ctypes.cdll.LoadLibrary('/lustre/orion/csc143/proj-shared/jwang/Comp4AI/ExternalDependencies/zfp/install/lib/python3.7/site-packages/zfpy.cpython-37m-x86_64-linux-gnu.so')
# Call a function from the library
my_array = np.arange(1, 20)
compressed_data = zfp.compress_numpy(my_array)
# Print the result
print("Result from my_function:", compressed_data)
The error message is
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
/tmp/ipykernel_276/1270200073.py in <module>
6 # Call a function from the library
7 my_array = np.arange(1, 20)
----> 8 compressed_data = zfp.compress_numpy(my_array)
9
10 # Print the result
/ccs/proj/csc143/jw447/jw-jupyter-env/lib/python3.7/ctypes/__init__.py in __getattr__(self, name)
375 if name.startswith('__') and name.endswith('__'):
376 raise AttributeError(name)
--> 377 func = self.__getitem__(name)
378 setattr(self, name, func)
379 return func
/ccs/proj/csc143/jw447/jw-jupyter-env/lib/python3.7/ctypes/__init__.py in __getitem__(self, name_or_ordinal)
380
381 def __getitem__(self, name_or_ordinal):
--> 382 func = self._FuncPtr((name_or_ordinal, self))
383 if not isinstance(name_or_ordinal, int):
384 func.__name__ = name_or_ordinal
AttributeError: /lustre/orion/csc143/proj-shared/jwang/Comp4AI/ExternalDependencies/zfp/install/lib/python3.7/site-packages/zfpy.cpython-37m-x86_64-linux-gnu.so: undefined symbol: compress_numpy
Could you please help me with the issue?
Thanks!
Any reason you can't just import zfpy? If that fails, you might have to set CMAKE_INSTALL_PREFIX to point where Python will find the .so. Where did make install put it?
Sorry for the absence. I attached the cmake install directory to LD_LIBRARY_PATH so that it contains
$proj_path/zfp/install/lib/python3.7/site-packages
$proj_path/zfp/install/lib64
Under site-packages, there is zfpy.cpython-37m-x86_64-linux-gnu.so. Under lib64, I have libzfp.so libzfp.so.1 libzfp.so.1.0.1
Then when I import zfpy, I am getting:
In [1]: import zfpy
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-1-a9d9756e3c1f> in <module>
----> 1 import zfpy
ImportError: libzfp.so.0: cannot open shared object file: No such file or directory
I am working with zfp 1.0.1.
And you've set LD_LIBRARY_PATH to point to $proj_path/zfp/install/lib64?
Yes, my LD_LIBRARY_PATH points to
$proj_path/zfp/install/lib64
$proj_path/zfp/install/lib/python3.7/site-packages
and
$proj_path/zfp/install
I'm at a loss as to what's going on here, but perhaps the following might provide some clues:
ls -l $proj_path/zfp/install/lib64
nm $proj_path/zfp/install/lib/python3.7/site-packages/zfpy.cpython-37m-x86_64-linux-gnu.so | grep zfp
ldd $proj_path/zfp/install/bin/zfp