aio-libs / frozenlist

`FrozenList` is a `list`-like structure that implements `collections.abc.MutableSequence` and can be made immutable.

Home Page:https://frozenlist.aio-libs.org

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

1.3.0: pep517 build fails

kloczek opened this issue · comments

Looks like pep517 based build using source tree fails

+ /usr/bin/python3 -sBm build -w --no-isolation
* Getting dependencies for wheel...
**********************
* Accellerated build *
**********************
running egg_info
creating frozenlist.egg-info
writing frozenlist.egg-info/PKG-INFO
writing dependency_links to frozenlist.egg-info/dependency_links.txt
writing top-level names to frozenlist.egg-info/top_level.txt
writing manifest file 'frozenlist.egg-info/SOURCES.txt'
listing git files failed - pretending there aren't any
reading manifest file 'frozenlist.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'frozenlist/*.c'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*.pyd' found anywhere in distribution
warning: no previously-included files matching '*.so' found anywhere in distribution
warning: no previously-included files matching '*.lib' found anywhere in distribution
warning: no previously-included files matching '*.dll' found anywhere in distribution
warning: no previously-included files matching '*.a' found anywhere in distribution
warning: no previously-included files matching '*.obj' found anywhere in distribution
warning: no previously-included files found matching 'frozenlist/*.html'
no previously-included directories found matching 'docs/_build'
adding license file 'LICENSE'
writing manifest file 'frozenlist.egg-info/SOURCES.txt'
* Building wheel...
**********************
* Accellerated build *
**********************
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-38
creating build/lib.linux-x86_64-cpython-38/frozenlist
copying frozenlist/__init__.py -> build/lib.linux-x86_64-cpython-38/frozenlist
running egg_info
writing frozenlist.egg-info/PKG-INFO
writing dependency_links to frozenlist.egg-info/dependency_links.txt
writing top-level names to frozenlist.egg-info/top_level.txt
listing git files failed - pretending there aren't any
reading manifest file 'frozenlist.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'frozenlist/*.c'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*.pyd' found anywhere in distribution
warning: no previously-included files matching '*.so' found anywhere in distribution
warning: no previously-included files matching '*.lib' found anywhere in distribution
warning: no previously-included files matching '*.dll' found anywhere in distribution
warning: no previously-included files matching '*.a' found anywhere in distribution
warning: no previously-included files matching '*.obj' found anywhere in distribution
warning: no previously-included files found matching 'frozenlist/*.html'
no previously-included directories found matching 'docs/_build'
adding license file 'LICENSE'
writing manifest file 'frozenlist.egg-info/SOURCES.txt'
copying frozenlist/__init__.pyi -> build/lib.linux-x86_64-cpython-38/frozenlist
copying frozenlist/_frozenlist.pyx -> build/lib.linux-x86_64-cpython-38/frozenlist
copying frozenlist/py.typed -> build/lib.linux-x86_64-cpython-38/frozenlist
running build_ext
building 'frozenlist._frozenlist' extension
creating build/temp.linux-x86_64-cpython-38
creating build/temp.linux-x86_64-cpython-38/frozenlist
/usr/bin/gcc -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none -D_GNU_SOURCE -fPIC -fwrapv -ffat-lto-objects -D_GNU_SOURCE -fPIC -fwrapv -O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none -D_GNU_SOURCE -fPIC -fwrapv -ffat-lto-objects -O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none -D_GNU_SOURCE -fPIC -fwrapv -ffat-lto-objects -O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none -fPIC -I/usr/include/python3.8 -c frozenlist/_frozenlist.c -o build/temp.linux-x86_64-cpython-38/frozenlist/_frozenlist.o
cc1: fatal error: frozenlist/_frozenlist.c: No such file or directory
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1

I can point on similar issue and solution against other module
fastavro/fastavro#600
fastavro/fastavro#602

Right, the pyproject.toml file doesn't state Cython is required. I'll see what I can do, perhaps today.

Right, the pyproject.toml file doesn't state Cython is required. I'll see what I can do, perhaps today.

It used to be in setup.py. It was a conscious choice to facilitate pure-Python builds in envs where one couldn't make build toolchain available easily. See also aio-libs/aiohttp#6760 (comment).

For prior discussions/problems, see

TL;DR The Python packaging ecosystem defaults aren't flexible enough for us to easily declare optional build dependencies. This is why the core aio-libs projects expect the users to call cythonize manually when they're building from the Git source or supply an explicit env var as an opt-out.
In the future, it should be possible to implement a PEP 517 in-tree build backend wrapping setuptools, for example, and returning additional dependencies based on the config that PEP 517 allows to specify (python -m build supports this). It'd be enought to define https://peps.python.org/pep-0517/#get-requires-for-build-sdist and https://peps.python.org/pep-0517/#get-requires-for-build-wheel, and proxy any other hook invocations to the setuptools' backend. Here's an example of a backend wrapping setuptools that I made a while back https://github.com/ansible/pylibssh/tree/devel/bin/pep517_backend, although it doesn't make use of config_settings.
Although, we may want to check what pip versions we'd like to support before such a backend is implemented.

Could you please try below:

git clone https://github.com/aio-libs/frozenlist
cd frozenlist
python3 -sBm build -w --no-isolation

You need to have installed build module.

This is not a supported build process.

Sorry to say that but using setup.py is already marked as deprecated 😞
Talks about switching to pep517 have been going for at least 4 years ..

That is not really correct. What is deprecated is invoking setup.py directly, outside the PEP 517 interface.

Nevertheless pep517 based build prcess still is not working.

+ /usr/bin/python3 -sBm build -w --no-isolation
* Getting build dependencies for wheel...
*********************
* Accelerated build *
*********************
running egg_info
creating frozenlist.egg-info
writing frozenlist.egg-info/PKG-INFO
writing dependency_links to frozenlist.egg-info/dependency_links.txt
writing top-level names to frozenlist.egg-info/top_level.txt
writing manifest file 'frozenlist.egg-info/SOURCES.txt'
listing git files failed - pretending there aren't any
reading manifest file 'frozenlist.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'frozenlist/*.c'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*.pyd' found anywhere in distribution
warning: no previously-included files matching '*.so' found anywhere in distribution
warning: no previously-included files matching '*.lib' found anywhere in distribution
warning: no previously-included files matching '*.dll' found anywhere in distribution
warning: no previously-included files matching '*.a' found anywhere in distribution
warning: no previously-included files matching '*.obj' found anywhere in distribution
warning: no previously-included files found matching 'frozenlist/*.html'
no previously-included directories found matching 'docs/_build'
adding license file 'LICENSE'
writing manifest file 'frozenlist.egg-info/SOURCES.txt'
* Building wheel...
*********************
* Accelerated build *
*********************
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-38
creating build/lib.linux-x86_64-cpython-38/frozenlist
copying frozenlist/__init__.py -> build/lib.linux-x86_64-cpython-38/frozenlist
running egg_info
writing frozenlist.egg-info/PKG-INFO
writing dependency_links to frozenlist.egg-info/dependency_links.txt
writing top-level names to frozenlist.egg-info/top_level.txt
listing git files failed - pretending there aren't any
reading manifest file 'frozenlist.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'frozenlist/*.c'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*.pyd' found anywhere in distribution
warning: no previously-included files matching '*.so' found anywhere in distribution
warning: no previously-included files matching '*.lib' found anywhere in distribution
warning: no previously-included files matching '*.dll' found anywhere in distribution
warning: no previously-included files matching '*.a' found anywhere in distribution
warning: no previously-included files matching '*.obj' found anywhere in distribution
warning: no previously-included files found matching 'frozenlist/*.html'
no previously-included directories found matching 'docs/_build'
adding license file 'LICENSE'
writing manifest file 'frozenlist.egg-info/SOURCES.txt'
copying frozenlist/__init__.pyi -> build/lib.linux-x86_64-cpython-38/frozenlist
copying frozenlist/_frozenlist.pyx -> build/lib.linux-x86_64-cpython-38/frozenlist
copying frozenlist/py.typed -> build/lib.linux-x86_64-cpython-38/frozenlist
running build_ext
building 'frozenlist._frozenlist' extension
creating build/temp.linux-x86_64-cpython-38
creating build/temp.linux-x86_64-cpython-38/frozenlist
/usr/bin/gcc -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none -D_GNU_SOURCE -fPIC -fwrapv -ffat-lto-objects -D_GNU_SOURCE -fPIC -fwrapv -O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none -D_GNU_SOURCE -fPIC -fwrapv -ffat-lto-objects -O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none -D_GNU_SOURCE -fPIC -fwrapv -ffat-lto-objects -O2 -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fdata-sections -ffunction-sections -flto=auto -flto-partition=none -fPIC -I/usr/include/python3.8 -c frozenlist/_frozenlist.c -o build/temp.linux-x86_64-cpython-38/frozenlist/_frozenlist.o
cc1: fatal error: frozenlist/_frozenlist.c: No such file or directory
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1

ERROR Backend subprocess exited when trying to invoke build_wheel

As linked above, the process is to cythonize first. Posting the same message multiple times will not change that explanation.

As linked above, the process is to cythonize first. Posting the same message multiple times will not change that explanation.

OK so why cythonize is not integrated like in all other projects and it must be called separately? 🤔

Because you can trivially install from the source distribution from PyPI, which includes the C files. It is only when you try to install from the GitHub repository that you need to run Cython.

Integrating into the 517 build process without setup.py is not currently in the cards because that's non-trivial, see the comment by webknjaz.

Because you can trivially install from the source distribution from PyPI, which includes the C files. It is only when you try to install from the GitHub repository that you need to run Cython.

I'm building from VCS tar ball to have guarantee that I would be able to apply any VCS patches added as commits after release.

Integrating into the 517 build process without setup.py is not currently in the cards because that's non-trivial, see the comment by webknjaz.

Just checked my python modules spec files and I see many modules in which I'm using VCS tar balls and on build process is used cython and all those modiels are using pep517 based build.

[tkloczko@pers-jacek SPECS]$ grep -l "BuildRequires:.*python3dist(cython)" python-* | xargs grep "^Source:.*VCS"
python-cftime.spec:Source:              %{VCS}/archive/v%{version}rel/%{name}-%{version}.tar.gz
python-clevercsv.spec:Source:           %{VCS}/archive/v%{version}/%{name}-%{version}.tar.gz
python-cpyparsing.spec:Source:          %{VCS}/archive/v%{version}/%{name}-%{version}.tar.gz
python-debugpy.spec:Source:             %{VCS}/archive/v%{version}/%{name}-%{version}.tar.gz
python-editdistance-s.spec:Source:              %{VCS}/archive/v%{version}/%{name}-%{version}.tar.gz
python-editdistance.spec:Source:                %{VCS}/archive/v%{version}/%{name}-%{version}.tar.gz
python-extension-helpers.spec:Source:           %{VCS}/archive/v%{version}/%{name}-%{version}.tar.gz
python-falcon.spec:Source:              %{VCS}/archive/%{version}/%{name}-%{version}.tar.gz
python-fastavro.spec:Source:            %{VCS}/archive/%{version}/%{name}-%{version}.tar.gz
python-gevent.spec:Source:              %{VCS}/archive/%{version}/%{name}-%{version}.tar.gz
python-gssapi.spec:Source:              %{VCS}/releases/download/v%{version}/%{name}-%{version}.tar.gz
python-hidapi.spec:Source:              %{VCS}/archive/%{version}/%{name}-%{version}.tar.gz
python-hunter.spec:Source:              %{VCS}/archive/v%{version}/%{name}-%{version}.tar.gz
python-ipython.spec:Source:             %{VCS}/archive/%(v=%{version}; echo ${v%~*})/%{name}-%(v=%{version}; echo ${v%~*}).tar.gz
python-logbook.spec:Source:             %{VCS}/archive/%{version}/%{name}-%{version}.tar.gz
python-lupa.spec:Source:                %{VCS}/archive/lupa-%{version}/%{name}-%{version}.tar.gz
python-lxml.spec:Source:                %{VCS}/archive/lxml-%{version}/%{name}-%{version}.tar.gz
python-mesonpy.spec:Source:             %{VCS}/archive/%{version}/%{name}-%{version}.tar.gz
python-mpi4py.spec:Source:              %{VCS}/archive/%{version}/%{name}-%{version}.tar.gz
python-msgpack.spec:Source:             %{VCS}/archive/v%{version}/%{name}-%{version}.tar.gz
python-multidict.spec:Source:           %{VCS}/archive/v%{version}/%{name}-%{version}.tar.gz
python-numpy.spec:Source:               %{VCS}/releases/download/v%{version}/numpy-%{version}.tar.gz
python-pandas.spec:Source:              %{VCS}/archive/v%{version}/%{name}-%{version}.tar.gz
python-pkcs11.spec:Source:              %{VCS}/archive/v%{version}/%{name}-%{version}.tar.gz
python-pydantic.spec:Source:            %{VCS}/archive/v%{version}/%{name}-%{version}.tar.gz
python-pyproj.spec:Source:              %{VCS}/archive/%{version}/%{name}-%{version}.tar.gz
python-pyyaml.spec:Source:              %{VCS}/archive/%{version}/%{name}-%{version}.tar.gz
python-rasterio.spec:Source:            %{VCS}/archive/%{version}/%{name}-%{version}.tar.gz
python-reedsolo.spec:Source:            %{VCS}/archive/v%{version}/%{name}-%{version}.tar.gz
python-scikit-build.spec:Source:                %{VCS}/archive/%{version}/%{name}-%{version}.tar.gz
python-shapely.spec:Source:             %{VCS}/archive/%{version}/%{name}-%{version}.tar.gz
python-sphinx-automodapi.spec:Source:           %{VCS}/archive/v%{version}/%{name}-%{version}.tar.gz
python-sqlalchemy.spec:Source:          %{VCS}/archive/rel_%(tr '.' '_'<<<%{version})/%{name}-%{version}.tar.gz
python-stack-data.spec:Source:          %{VCS}/archive/v%{version}/%{name}-%{version}.tar.gz
python-sympy.spec:Source:               %{VCS}//archive/sympy-%{version}/%{name}-%{version}.tar.gz
python-twisted.spec:Source:             %{VCS}/archive/twisted-%{version}/%{name}-%{version}.tar.gz
python-uvloop.spec:Source:              %{VCS}/archive/v%{version}/%{name}-%{version}.tar.gz
python-yarl.spec:Source:                %{VCS}/archive/v%{version}/%{name}-%{version}.tar.gz
python-zeroconf.spec:Source:            %{VCS}/archive/%{version}/%{name}-%{version}.tar.gz
python-zmq.spec:Source:         %{VCS}/archive/v%{version}/%{name}-%{version}.tar.gz

Here is a bit shorter list of VCS URLs of such modules which are using setuptools and cython.

[tkloczko@pers-jacek SPECS]$ grep -l "BuildRequires:.*python3dist(cython)" python-* | xargs grep "^Source:.*VCS" -l | xargs grep "BuildRequires:.*python3dist(setuptools)" -l | xargs grep ^VCS
python-cftime.spec:VCS:         https://github.com/Unidata/cftime/
python-clevercsv.spec:VCS:              https://github.com/alan-turing-institute/CleverCSV/
python-cpyparsing.spec:VCS:             https://github.com/evhub/cpyparsing/
python-debugpy.spec:VCS:                https://github.com/microsoft/debugpy/
python-editdistance-s.spec:VCS:         https://github.com/asottile/editdistance-s/
python-editdistance.spec:VCS:           https://github.com/roy-ht/editdistance/
python-falcon.spec:VCS:         https://github.com/falconry/falcon/
python-fastavro.spec:VCS:               https://github.com/fastavro/fastavro/
python-gevent.spec:VCS:         https://github.com/gevent/gevent/
python-gssapi.spec:VCS:         https://github.com/pythongssapi/python-gssapi/
python-hidapi.spec:VCS:         https://github.com/trezor/cython-hidapi/
python-hunter.spec:VCS:         https://github.com/ionelmc/python-hunter/
python-ipython.spec:VCS:                https://github.com/ipython/ipython/
python-logbook.spec:VCS:                https://github.com/getlogbook/logbook/
python-lupa.spec:VCS:           https://github.com/scoder/lupa/
python-lxml.spec:VCS:           https://github.com/lxml/lxml/
python-mesonpy.spec:VCS:                https://github.com/mesonbuild/meson-python/
python-mpi4py.spec:VCS:         https://github.com/mpi4py/mpi4py/
python-msgpack.spec:VCS:                https://github.com/msgpack/msgpack-python/
python-multidict.spec:VCS:              https://github.com/aio-libs/multidict/
python-numpy.spec:VCS:          https://github.com/numpy/numpy/
python-pandas.spec:VCS:         https://github.com/pandas-dev/pandas/
python-pydantic.spec:VCS:               https://github.com/samuelcolvin/pydantic/
python-pyproj.spec:VCS:         https://github.com/pyproj4/pyproj/
python-pyyaml.spec:VCS:         https://github.com/yaml/pyyaml/
python-rasterio.spec:VCS:               https://github.com/mapbox/rasterio/
python-reedsolo.spec:VCS:               https://github.com/tomerfiliba/reedsolomon/
python-shapely.spec:VCS:                https://github.com/shapely/shapely/
python-sphinx-automodapi.spec:VCS:              https://github.com/astropy/sphinx-automodapi/
python-sqlalchemy.spec:VCS:             https://github.com/sqlalchemy/sqlalchemy/
python-sympy.spec:VCS:          https://github.com/sympy/sympy/
python-twisted.spec:VCS:                https://github.com/twisted/twisted/
python-uvloop.spec:VCS:         https://github.com/MagicStack/uvloop/
python-yarl.spec:VCS:           https://github.com/aio-libs/yarl/
python-zeroconf.spec:VCS:               https://github.com/jstasiak/python-zeroconf/
python-zmq.spec:VCS:            https://github.com/zeromq/pyzmq/

We already had this conversation. "because somebody else does something" is not a good argument for doing the same.

Integrating into the 517 build process without setup.py is not currently in the cards because that's non-trivial, see the comment by webknjaz.

It's now implemented on master, similarly to yarl.

Thx.
As it is more 100 commits since last release IMO it would be good to try release new version.
Is it anything else on outstanding list? 🤔