cloud-init upstream setup.py packaging is incompatible with PEP517 distro implementations
holmanb opened this issue · comments
Bug report
tl;dr: Changing cloud-init's packaging paradigm to support PEP517 packaging tooling appears unavoidable to support PEP517.
In a476ecb we tried switching debian packaging from dh-python's pybuild distutils plugin, (which depends on a standard library package which was removed from stdlib in 3.12) to dh-python's pybuild pyproject plugin.
After this change, tests on this branch started failing due to ds-identify and other "data_files" being installed in the wrong location.
After filing a bug with debian against dh-python, I had an IRC conversation in #debian-python on Libera. To summarize the conversation: other non-debian distros' packaging tooling similarly does wheel-based installations, so it's unlikely that we'll be able to keep the current packaging paradigm.
Debian's interpretation here is that data should be installed into PREFIX (i.e. /usr). This matches what pip would do, and lines up with virtualenvs, which can be looked at as a different PREFIX.
So, yes, you can help distributions by laying out data usefully, but in Debian's case, we'll probably need to move the data around after we've installed the wheel.
I had discussed this in the #cloud-init IRC channel several months ago as Alpine had switched a lot of packages from setup.py to gpep517.
After this change, tests on this branch started failing due to ds-identify and other "data_files" being installed in the wrong location.
That was exactly the same issue I experienced then when I tried to switch the Alpine cloud-init package to gpep517. At the time (from memory) Sam (_sam on IRC) indicated that gpep517 didn't support files outside of /var/lib/python/site-packages/ currently
I had discussed this in the #cloud-init IRC channel several months ago as Alpine had switched a lot of packages from setup.py to gpep517.
After this change, tests on this branch started failing due to ds-identify and other "data_files" being installed in the wrong location.
That was exactly the same issue I experienced then when I tried to switch the Alpine cloud-init package to gpep517. At the time (from memory) Sam (_sam on IRC) indicated that gpep517 didn't support files outside of /var/lib/python/site-packages/ currently
@dermotbradley +1 Thanks for adding the extra context.
The python ecosystem officially considers it unsupported to use python packaging technology for anything other than:
- *.py importable libraries of code
- command-line entry scripts
The wheel standard supports installing files to hardcoded locations inside of /usr if and only if sys.prefix for python itself is /usr -- this means that installing into a virtualenv is probably broken as I somehow doubt the software will work correctly when installing /home/myname/.venvmanager/venvs/cloud-init/lib/udev/*, at which point you really, really should be asking "what am I getting out of this".
Note also that it is IMPOSSIBLE to install files to /etc using a wheel, since /etc inherently lies outside of python's sys.prefix.
On the flip side, you need to ask what you actually get out of wheels. It allows users to pip install the software, but do you need that functionality? Probably not... just get the software from the distro repository.
Effectively, python packaging is being used as a glorified makefile, and setup.py is kind of convenient because it supports locating the system site-packages directory.
I would recommend ditching the python packaging ecosystem entirely if you don't need it, and moving to a real build system such as meson.
You still get support for installing python modules using py_installation.install_sources(), but you also get to install files to standard locations such as --prefix, --sysconfdir, --localstatedir, and so on.