Cross-compiling result in ImportError: libffi.so.8: cannot open shared object file: No such file or directory
th0ma7 opened this issue · comments
While preparing an update for our python-3.8 package for Synology NAS (uses linux called DSM), the CI was able to cross-compile all packages and dependencies with the only exception, cross-compiling x86_64 (all builds are done from a x86_64 host docker image). My initial thoughts are that it somehow loose track of the library path, hence trying to find libffi.so
from the host default path (but I may well be wrong).
PR is SynoCommunity/spksrc#4902 and last CI run is https://github.com/SynoCommunity/spksrc/pull/4902/checks?check_run_id=3858396685 where it failed only on x64 arch for DSM version 6.1 and 7.0. All packages are built using Synology toolchain for each associated architectures.
Using version 1.4.0 from https://pypi.org/project/PyNaCl/ or latest snapshot from github gets to the exact same error.
Cross-compiling using following versions:
python
: 3.8.12libffi
: 3.4.2cffi
(pip/wheel): 1.14.6, 1.14.5PyNaCl
: 1.3.0, 1.4.0, github-master
End of the build error:
make[4]: Leaving directory '/home/spksrc/python38-update-fix/spksrc/cross/libsodium'
===> Extracting for PyNaCl
tar -xzpf /home/spksrc/python38-update-fix/spksrc/cross/PyNaCl/../../distrib/PyNaCl-gita5a7876b523056241b53a49b7e53e7de0d8e7ccd.tar.gz -C /home/spksrc/python38-update-fix/spksrc/spk/homeassistant/work-x64-7.0
===> Patching for PyNaCl
===> Configuring for PyNaCl
===> Compiling for PyNaCl
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "setup.py", line 203, in <module>
setup(
File "/home/spksrc/python38-update-fix/spksrc/spk/homeassistant/work-x64-7.0/install/var/packages/homeassistant/target/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
return distutils.core.setup(**attrs)
File "/home/spksrc/python38-update-fix/spksrc/native/python38/work-native/install/usr/local/lib/python3.8/distutils/core.py", line 108, in setup
_setup_distribution = dist = klass(attrs)
File "/home/spksrc/python38-update-fix/spksrc/spk/homeassistant/work-x64-7.0/install/var/packages/homeassistant/target/lib/python3.8/site-packages/setuptools/dist.py", line 453, in __init__
_Distribution.__init__(
File "/home/spksrc/python38-update-fix/spksrc/native/python38/work-native/install/usr/local/lib/python3.8/distutils/dist.py", line 292, in __init__
self.finalize_options()
File "/home/spksrc/python38-update-fix/spksrc/spk/homeassistant/work-x64-7.0/install/var/packages/homeassistant/target/lib/python3.8/site-packages/setuptools/dist.py", line 831, in finalize_options
ep(self)
File "/home/spksrc/python38-update-fix/spksrc/spk/homeassistant/work-x64-7.0/install/var/packages/homeassistant/target/lib/python3.8/site-packages/setuptools/dist.py", line 852, in _finalize_setup_keywords
ep.load()(self, ep.name, value)
File "/home/spksrc/python38-update-fix/spksrc/spk/homeassistant/work-x64-7.0/install/var/packages/homeassistant/target/lib/python3.8/site-packages/cffi/setuptools_ext.py", line 219, in cffi_modules
add_cffi_module(dist, cffi_module)
File "/home/spksrc/python38-update-fix/spksrc/spk/homeassistant/work-x64-7.0/install/var/packages/homeassistant/target/lib/python3.8/site-packages/cffi/setuptools_ext.py", line 49, in add_cffi_module
execfile(build_file_name, mod_vars)
File "/home/spksrc/python38-update-fix/spksrc/spk/homeassistant/work-x64-7.0/install/var/packages/homeassistant/target/lib/python3.8/site-packages/cffi/setuptools_ext.py", line 25, in execfile
exec(code, glob, glob)
File "src/bindings/build.py", line 35, in <module>
ffi = FFI()
File "/home/spksrc/python38-update-fix/spksrc/spk/homeassistant/work-x64-7.0/install/var/packages/homeassistant/target/lib/python3.8/site-packages/cffi/api.py", line 48, in __init__
import _cffi_backend as backend
ImportError: libffi.so.8: cannot open shared object file: No such file or directory
make[3]: *** [../../mk/spksrc.python-wheel.mk:33: build_python_wheel] Error 1
make[3]: Leaving directory '/home/spksrc/python38-update-fix/spksrc/cross/PyNaCl'
make[2]: *** [../../mk/spksrc.depend.mk:51: depend_target] Error 2
make[2]: Leaving directory '/home/spksrc/python38-update-fix/spksrc/spk/homeassistant'
make[1]: Leaving directory '/home/spksrc/python38-update-fix/spksrc/spk/homeassistant'
While the libraries are available:
$ find . -name libffi.so.8
./work-x64-7.0/install/var/packages/homeassistant/target/lib/libffi.so.8
./work-x64-7.0/libffi-3.4.2/x86_64-pc-linux-gnu/.libs/libffi.so.8
./work-x64-6.1/libffi-3.4.2/x86_64-pc-linux-gnu/.libs/libffi.so.8
./work-x64-6.1/install/var/packages/homeassistant/target/lib/libffi.so.8
And the work-x64-<DSM-version>/install/var/packages/homeassistant/target/lib
is part of the library path.
This is a cross-compile error with cffi
, which pyNaCl does depend on, but is not something we control.
I was able to found the reason why: crossenv
Our build framework starts by building a minimal python38
and includes the necessary wheels, including crossenv
.
It happens that all version >=1.0 of crossenv
makes PyNaCl to fail to build in our cross-compiling environment, only for x86_64
architecture.
I presume that crossenv
has a bug that gets triggered in cross-compiling environment where if the target is of the same arch of the host it goes outside of its boundary instead of using the passed library paths of our toolchain.