Include Python development headers

[Jakub Kuczys]

Describe the Enhancement

Currently, the buildpack does not contain Python development headers which makes it hard to install any Python packages implemented in C that don't provide wheels for the platform. One such package is psycopg2 which purposefully doesn't provide source distribution to allow usage of system's own libraries on production systems.

In order to see the problem, you can run these commands:

git clone https://github.com/paketo-buildpacks/samples
cd samples/python/pip
echo "psycopg2==2.9.5" >> requirements.txt
pack build pip-sample --buildpack paketo-buildpacks/python --builder paketobuildpacks/builder:full

which results in an error about missing Python headers:

      ./psycopg/psycopg.h:35:10: fatal error: Python.h: No such file or directory
       #include <Python.h>
                ^~~~~~~~~~
      compilation terminated.

Possible Solution

Include development headers in appropriate environments (potential candidates: CPATH, CPPPATH, PKG_CONFIG_PATH, INCLUDE_PATH) or symlink them into one of the default include directories (not sure that will work properly).

Motivation

The lack of development headers makes it hard to install any Python packages implemented in C (C extensions) that don't provide wheels for the platform. One such package is psycopg2 which purposefully doesn't provide source distribution to allow usage of system's own libraries on production systems.

[Rob Dimsdale-Zucker]

@Jackenmen thanks for reporting this. I agree it's an issue. Thank you for provide a clear, simple reproduction. I was able to reproduce the issue quickly.

I'll prioritize this for us to fix, and keep you updated with our progress.

---

For maintainers' reference, the header files are present on the resultant image, so the pre-compiled CPython dependency is probably fine. It's more likely that we need to set/append an environment variable to point to the correct location (as suggested above).

git clone https://github.com/paketo-buildpacks/samples
cd samples/python/pip
# no modifications to the requirements.txt file
pack build pip-sample --buildpack paketo-buildpacks/python --builder paketobuildpacks/builder:full
docker run --rm -it --entrypoint launcher pip-sample ls -alh /layers/paketo-buildpacks_cpython/cpython/include/python3.10/Python.h
-rw-r--r-- 1 cnb cnb 3.2K Jan  1  1980 /layers/paketo-buildpacks_cpython/cpython/include/python3.10/Python.h

Also for reference, this is the run-time environment using the same app image as above. It might be slightly different from the build-time environment, but should be a useful starting-off point.

docker run --rm -it --entrypoint launcher pip-sample env
LD_LIBRARY_PATH=/layers/paketo-buildpacks_pip-install/packages/lib:/layers/paketo-buildpacks_cpython/cpython/lib
HOSTNAME=00e87b622780
OLDPWD=/workspace
PWD=/workspace
HOME=/home/cnb
PYTHONPYCACHEPREFIX=/home/cnb/.pycache
TERM=xterm
SHLVL=1
PYTHONPATH=/layers/paketo-buildpacks_pip-install/packages/lib/python3.10/site-packages:/layers/paketo-buildpacks_cpython/cpython
PATH=/layers/paketo-buildpacks_pip-install/packages/bin:/layers/paketo-buildpacks_cpython/cpython/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
_=/usr/bin/env

[Ed Morley]

Btw this is likely due to pypa/setuptools#3657 which was a setuptools regression in header autodetection, which first appeared in setuptools 65.2.0 and was fixed in setuptools 67.2.0. (Though I haven't tested the Paketo Python CNB to confirm.)

The Heroku Python CNB currently explicitly sets CPATH to work around the issue (though it's less important now that the issue has been fixed in setuptools):
https://github.com/heroku/buildpacks-python/blob/e618353b041f00c4b57201ecfbcfd9b69d3fcc58/src/layers/python.rs#L270-L283