Cmake tries to install to `/usr/local/lib/python3.11/site-packages` instead of the local environment

[Julian Hossbach]

Hey,
I am currently trying to make the espresso MD package pip-installable. I had successfully installed the package using scikit-build with the following setup.py:

from skbuild import setup

setup(
    name="espressomd",
    version="4.3",
    description="Extensible Simulation Package for Research on Soft Matter Systems",
    author="The ESPResSo project",
    url="https://espressomd.org",
    license_files=["COPYING"],
)

But now, when moving to scikit-build-core, I ran into the following issue

 scikit_build_core - INFO - RUN: /tmp/pip-build-env-tv8rqy58/overlay/lib/python3.11/site-packages/cmake/data/bin/cmake --install /tmp/tmpkzta0wja/build --prefix /tmp/tmpkzta0wja/wheel/platlib --strip
  -- Install configuration: "Release"
 CMake Error at /tmp/tmpkzta0wja/build/src/shapes/cmake_install.cmake:60 (file):
    file cannot create directory:
    /usr/local/lib/python3.11/site-packages/espressomd.  Maybe need
    administrative privileges.

This is the pyproject.toml:

[build-system]
requires = [
  "scikit-build-core",
  "cmake>=3.18",
  "numpy>=1.23",
  "cython>=0.29.21,<=3.0.7",
]
build-backend = "scikit_build_core.build"

[project]
name = "espressomd"
version = "4.3"

This is the output of python -m scikit_build_core.builder:

Scikit-build-core 0.8.2 on Python 3.11.8 (main, Feb 12 2024, 14:50:05) [GCC 13.2.1 20230801]
Detected Python Library: /usr/lib/libpython3.11.so
Detected ABI3 Python Library: /usr/lib/libpython3.11.so
Detected Python Include Directory: /usr/include/python3.11
Detected NumPy Include Directory: None
Detected Platform: linux-x86_64
Detected SOABI: cpython-311-x86_64-linux-gnu
Detected ABI3 SOABI: abi3
Default Wheel Tag: cp311-cp311-manylinux_2_39_x86_64
 - Note: use python -m scikit_build_core.builder.wheel_tag for further options
Get Requires: ['cmake>=3.15']
Detected CMake and Ninja (all versions):
* CMake: /usr/bin/cmake <Version('3.28.4')>
* Ninja: /usr/bin/ninja <Version('1.11.1')>

Any ideas on what the problem is?

[Cristian Le]

Issue is with ESPRESSO_INSTALL_PYTHON and the other variables used in install in the cmake files. These should point to ./ see https://scikit-build-core.readthedocs.io/en/latest/cmakelists.html#install-directories

[Henry Schreiner]

I think that’s here: https://github.com/espressomd/espresso/blob/7ad0534debbbe9dac048f4ff322054001de24587/CMakeLists.txt#L266

It’s trying to install into site packages directly instead of the wheel.

[Julian Hossbach]

Yeah, setting ESPRESSO_INSTALL_PYTHON to ./ did solve the failing build process, but it cannot find a shared file (espresso_script_interface.so) when trying to import. However, directly importing the package from within the venvs lib/python3.11/site-packages/ works, any ideas on that?

[Cristian Le]

That is mostly dependent on how the libraries are linked via RPATH (in the build directory it should be linked appropriately that you can execute them), and after install when the RPATH is stripped, it depends on what library paths are visible, e.g. via LD_LIBRARY_PATH. The main issue here is where is espresso_script_interface.so installed to? You can use CMAKE_INSTALL_RPATH if you know in advance.

There are 2 more designs you can consider, making espresso_script_interface.a a static library, or having the python script pre-load the library from a known location. It ultimately depends on the details of the project which approach is most suitable

[Julian Hossbach]

Thanks a lot for the helpful input! I will discuss this with the maintainers of espresso and leave this issue open until we have come to a conclusion.

[Julian Hossbach]

I was able to resolve this issue by properly setting the RPATH of the shared object files using CMakes Development environment, now every .so is found. Thanks a lot for the help!