Issue with build/ directory with pybuild build system
carnil opened this issue · comments
Hi
In Debian I'm using the pybuild system (which invokes the build systems for requested Python versions). I noticed, when the request happends for multiple version, say
pybuild --build -i python{version} -p "3.11 3.10 3.9"
and all three version are supported, then the build directory in the end contains multiple recursions of the lib directory, e.g.
build/lib/cvelib/cli.py <-- first run with 3.11
build/lib/build/lib/cvelib/cli.py <-- second run with 3.10
build/lib/build/lib/build/lib/cvelib/cli.py <-- third run with 3.9
pybuild --build -i python{version} -p "3.11 3.10 3.9" 2>&1 | grep '/cli.py' | grep copying
copying cvelib/cli.py -> build/lib/cvelib
copying build/lib/cvelib/cli.py -> build/lib/build/lib/cvelib
copying build/lib/cvelib/cli.py -> build/bdist.linux-x86_64/wheel/cvelib
copying build/lib/build/lib/cvelib/cli.py -> build/bdist.linux-x86_64/wheel/build/lib/cvelib
copying build/lib/build/lib/cvelib/cli.py -> build/lib/build/lib/build/lib/cvelib
copying build/lib/cvelib/cli.py -> build/bdist.linux-x86_64/wheel/cvelib
copying build/lib/build/lib/cvelib/cli.py -> build/bdist.linux-x86_64/wheel/build/lib/cvelib
copying build/lib/build/lib/build/lib/cvelib/cli.py -> build/bdist.linux-x86_64/wheel/build/lib/build/lib/cvelib
I suspect there is somethng wrong with the build system of cvelib recently around 7dfa855 . Do you see what the problem might be?
I assume patches welcome, but I have not the solution yet on what is actually going wrong.
--- a/setup.py
+++ b/setup.py
@@ -61,7 +61,7 @@ setup(
"Programming Language :: Python :: 3.11",
],
include_package_data=True,
- packages=find_namespace_packages(),
+ packages=["cvelib"],
install_requires=install_requires,
extras_require=extras_require,
entry_points={
would partially solve the problem I encountered, but is probably not what you would like to have, as the manpages man are then not installed as you wanted it (is though site-packages correct?) and tests (but is this as well wanted?).
Hmm, this looks like a bug with your pybuild build system. When it builds each package for a different version of Python, it should start with a clean environment. In your output it seems the 3.10 build runs in the same directory where the 3.9 build already completed, so when the find_namespace_packages() function runs, it includes the build directory as part of the cvelib package itself. The point of using this function is to pick up the cvelib.schemas directory since it's not a proper Python module (doesn't include an empty __init__.py file), but it should still be included in the package. The inclusion of tests and man pages is also deliberate.
That being said, I couldn't reproduce this on my own system:
root@ba1358ae4029:/tmp/cvelib# pybuild --build -i python{version} -p "3.11 3.10 3.9"
I: pybuild base:232: /usr/bin/python3.10 setup.py build
/usr/local/lib/python3.10/site-packages/setuptools/dist.py:697: UserWarning: Usage of dash-separated 'build-lib' will not be supported in future versions. Please use the underscore name 'build_lib' instead
warnings.warn(
/usr/local/lib/python3.10/site-packages/setuptools/dist.py:697: UserWarning: Usage of dash-separated 'install-layout' will not be supported in future versions. Please use the underscore name 'install_layout' instead
warnings.warn(
/usr/local/lib/python3.10/site-packages/setuptools/dist.py:697: UserWarning: Usage of dash-separated 'install-scripts' will not be supported in future versions. Please use the underscore name 'install_scripts' instead
warnings.warn(
/usr/local/lib/python3.10/site-packages/setuptools/dist.py:697: UserWarning: Usage of dash-separated 'install-lib' will not be supported in future versions. Please use the underscore name 'install_lib' instead
warnings.warn(
running build
running build_py
creating /tmp/cvelib/.pybuild/cpython3_3.10/build/cvelib
copying cvelib/__init__.py -> /tmp/cvelib/.pybuild/cpython3_3.10/build/cvelib
copying cvelib/cli.py -> /tmp/cvelib/.pybuild/cpython3_3.10/build/cvelib
copying cvelib/cve_api.py -> /tmp/cvelib/.pybuild/cpython3_3.10/build/cvelib
package init file 'man/__init__.py' not found (or not a regular file)
package init file 'tests/__init__.py' not found (or not a regular file)
creating /tmp/cvelib/.pybuild/cpython3_3.10/build/tests
copying tests/test_cli.py -> /tmp/cvelib/.pybuild/cpython3_3.10/build/tests
copying tests/test_cve_api.py -> /tmp/cvelib/.pybuild/cpython3_3.10/build/tests
package init file 'cvelib/schemas/__init__.py' not found (or not a regular file)
creating /tmp/cvelib/.pybuild/cpython3_3.10/build/cvelib/schemas
copying cvelib/schemas/extract_container_schemas.py -> /tmp/cvelib/.pybuild/cpython3_3.10/build/cvelib/schemas
package init file 'tests/data/__init__.py' not found (or not a regular file)
running egg_info
writing cvelib.egg-info/PKG-INFO
writing dependency_links to cvelib.egg-info/dependency_links.txt
writing entry points to cvelib.egg-info/entry_points.txt
writing requirements to cvelib.egg-info/requires.txt
writing top-level names to cvelib.egg-info/top_level.txt
reading manifest file 'cvelib.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '__pycache__' found anywhere in distribution
warning: no previously-included files matching '*.py[co]' found anywhere in distribution
adding license file 'LICENSE'
writing manifest file 'cvelib.egg-info/SOURCES.txt'
creating /tmp/cvelib/.pybuild/cpython3_3.10/build/man
copying man/cve-list.1 -> /tmp/cvelib/.pybuild/cpython3_3.10/build/man
copying man/cve-org-users.1 -> /tmp/cvelib/.pybuild/cpython3_3.10/build/man
copying man/cve-org.1 -> /tmp/cvelib/.pybuild/cpython3_3.10/build/man
copying man/cve-ping.1 -> /tmp/cvelib/.pybuild/cpython3_3.10/build/man
copying man/cve-publish.1 -> /tmp/cvelib/.pybuild/cpython3_3.10/build/man
copying man/cve-quota.1 -> /tmp/cvelib/.pybuild/cpython3_3.10/build/man
copying man/cve-reject.1 -> /tmp/cvelib/.pybuild/cpython3_3.10/build/man
copying man/cve-reserve.1 -> /tmp/cvelib/.pybuild/cpython3_3.10/build/man
copying man/cve-show.1 -> /tmp/cvelib/.pybuild/cpython3_3.10/build/man
copying man/cve-undo-reject.1 -> /tmp/cvelib/.pybuild/cpython3_3.10/build/man
copying man/cve-user-create.1 -> /tmp/cvelib/.pybuild/cpython3_3.10/build/man
copying man/cve-user-reset-key.1 -> /tmp/cvelib/.pybuild/cpython3_3.10/build/man
copying man/cve-user-update.1 -> /tmp/cvelib/.pybuild/cpython3_3.10/build/man
copying man/cve-user.1 -> /tmp/cvelib/.pybuild/cpython3_3.10/build/man
copying man/cve.1 -> /tmp/cvelib/.pybuild/cpython3_3.10/build/man
copying cvelib/schemas/CVE_JSON_5.0_bundled_5.0.0.json -> /tmp/cvelib/.pybuild/cpython3_3.10/build/cvelib/schemas
copying cvelib/schemas/README.md -> /tmp/cvelib/.pybuild/cpython3_3.10/build/cvelib/schemas
copying cvelib/schemas/adp_container_5.0.0.json -> /tmp/cvelib/.pybuild/cpython3_3.10/build/cvelib/schemas
copying cvelib/schemas/published_cna_container_5.0.0.json -> /tmp/cvelib/.pybuild/cpython3_3.10/build/cvelib/schemas
copying cvelib/schemas/rejected_cna_container_5.0.0.json -> /tmp/cvelib/.pybuild/cpython3_3.10/build/cvelib/schemas
creating /tmp/cvelib/.pybuild/cpython3_3.10/build/tests/data
copying tests/data/CVEv5_advanced-example.json -> /tmp/cvelib/.pybuild/cpython3_3.10/build/tests/data
copying tests/data/CVEv5_basic-example.json -> /tmp/cvelib/.pybuild/cpython3_3.10/build/tests/data
copying tests/data/README.md -> /tmp/cvelib/.pybuild/cpython3_3.10/build/tests/data
I: pybuild base:232: /usr/bin/python3.11 setup.py build
/usr/local/lib/python3.11/site-packages/setuptools/dist.py:771: UserWarning: Usage of dash-separated 'build-lib' will not be supported in future versions. Please use the underscore name 'build_lib' instead
warnings.warn(
/usr/local/lib/python3.11/site-packages/setuptools/dist.py:771: UserWarning: Usage of dash-separated 'install-layout' will not be supported in future versions. Please use the underscore name 'install_layout' instead
warnings.warn(
/usr/local/lib/python3.11/site-packages/setuptools/dist.py:771: UserWarning: Usage of dash-separated 'install-scripts' will not be supported in future versions. Please use the underscore name 'install_scripts' instead
warnings.warn(
/usr/local/lib/python3.11/site-packages/setuptools/dist.py:771: UserWarning: Usage of dash-separated 'install-lib' will not be supported in future versions. Please use the underscore name 'install_lib' instead
warnings.warn(
running build
running build_py
creating /tmp/cvelib/.pybuild/cpython3_3.11/build/cvelib
copying cvelib/__init__.py -> /tmp/cvelib/.pybuild/cpython3_3.11/build/cvelib
copying cvelib/cli.py -> /tmp/cvelib/.pybuild/cpython3_3.11/build/cvelib
copying cvelib/cve_api.py -> /tmp/cvelib/.pybuild/cpython3_3.11/build/cvelib
creating /tmp/cvelib/.pybuild/cpython3_3.11/build/tests
copying tests/test_cli.py -> /tmp/cvelib/.pybuild/cpython3_3.11/build/tests
copying tests/test_cve_api.py -> /tmp/cvelib/.pybuild/cpython3_3.11/build/tests
creating /tmp/cvelib/.pybuild/cpython3_3.11/build/cvelib/schemas
copying cvelib/schemas/extract_container_schemas.py -> /tmp/cvelib/.pybuild/cpython3_3.11/build/cvelib/schemas
running egg_info
writing cvelib.egg-info/PKG-INFO
writing dependency_links to cvelib.egg-info/dependency_links.txt
writing entry points to cvelib.egg-info/entry_points.txt
writing requirements to cvelib.egg-info/requires.txt
writing top-level names to cvelib.egg-info/top_level.txt
reading manifest file 'cvelib.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '__pycache__' found anywhere in distribution
warning: no previously-included files matching '*.py[co]' found anywhere in distribution
adding license file 'LICENSE'
writing manifest file 'cvelib.egg-info/SOURCES.txt'
creating /tmp/cvelib/.pybuild/cpython3_3.11/build/man
copying man/cve-list.1 -> /tmp/cvelib/.pybuild/cpython3_3.11/build/man
copying man/cve-org-users.1 -> /tmp/cvelib/.pybuild/cpython3_3.11/build/man
copying man/cve-org.1 -> /tmp/cvelib/.pybuild/cpython3_3.11/build/man
copying man/cve-ping.1 -> /tmp/cvelib/.pybuild/cpython3_3.11/build/man
copying man/cve-publish.1 -> /tmp/cvelib/.pybuild/cpython3_3.11/build/man
copying man/cve-quota.1 -> /tmp/cvelib/.pybuild/cpython3_3.11/build/man
copying man/cve-reject.1 -> /tmp/cvelib/.pybuild/cpython3_3.11/build/man
copying man/cve-reserve.1 -> /tmp/cvelib/.pybuild/cpython3_3.11/build/man
copying man/cve-show.1 -> /tmp/cvelib/.pybuild/cpython3_3.11/build/man
copying man/cve-undo-reject.1 -> /tmp/cvelib/.pybuild/cpython3_3.11/build/man
copying man/cve-user-create.1 -> /tmp/cvelib/.pybuild/cpython3_3.11/build/man
copying man/cve-user-reset-key.1 -> /tmp/cvelib/.pybuild/cpython3_3.11/build/man
copying man/cve-user-update.1 -> /tmp/cvelib/.pybuild/cpython3_3.11/build/man
copying man/cve-user.1 -> /tmp/cvelib/.pybuild/cpython3_3.11/build/man
copying man/cve.1 -> /tmp/cvelib/.pybuild/cpython3_3.11/build/man
copying cvelib/schemas/CVE_JSON_5.0_bundled_5.0.0.json -> /tmp/cvelib/.pybuild/cpython3_3.11/build/cvelib/schemas
copying cvelib/schemas/README.md -> /tmp/cvelib/.pybuild/cpython3_3.11/build/cvelib/schemas
copying cvelib/schemas/adp_container_5.0.0.json -> /tmp/cvelib/.pybuild/cpython3_3.11/build/cvelib/schemas
copying cvelib/schemas/published_cna_container_5.0.0.json -> /tmp/cvelib/.pybuild/cpython3_3.11/build/cvelib/schemas
copying cvelib/schemas/rejected_cna_container_5.0.0.json -> /tmp/cvelib/.pybuild/cpython3_3.11/build/cvelib/schemas
creating /tmp/cvelib/.pybuild/cpython3_3.11/build/tests/data
copying tests/data/CVEv5_advanced-example.json -> /tmp/cvelib/.pybuild/cpython3_3.11/build/tests/data
copying tests/data/CVEv5_basic-example.json -> /tmp/cvelib/.pybuild/cpython3_3.11/build/tests/data
copying tests/data/README.md -> /tmp/cvelib/.pybuild/cpython3_3.11/build/tests/data
I: pybuild base:232: /usr/bin/python3 setup.py build
running build
running build_py
creating /tmp/cvelib/.pybuild/cpython3_3.9/build/cvelib
copying cvelib/__init__.py -> /tmp/cvelib/.pybuild/cpython3_3.9/build/cvelib
copying cvelib/cli.py -> /tmp/cvelib/.pybuild/cpython3_3.9/build/cvelib
copying cvelib/cve_api.py -> /tmp/cvelib/.pybuild/cpython3_3.9/build/cvelib
package init file 'man/__init__.py' not found (or not a regular file)
package init file 'tests/__init__.py' not found (or not a regular file)
creating /tmp/cvelib/.pybuild/cpython3_3.9/build/tests
copying tests/test_cli.py -> /tmp/cvelib/.pybuild/cpython3_3.9/build/tests
copying tests/test_cve_api.py -> /tmp/cvelib/.pybuild/cpython3_3.9/build/tests
package init file 'cvelib/schemas/__init__.py' not found (or not a regular file)
creating /tmp/cvelib/.pybuild/cpython3_3.9/build/cvelib/schemas
copying cvelib/schemas/extract_container_schemas.py -> /tmp/cvelib/.pybuild/cpython3_3.9/build/cvelib/schemas
package init file 'tests/data/__init__.py' not found (or not a regular file)
running egg_info
writing cvelib.egg-info/PKG-INFO
writing dependency_links to cvelib.egg-info/dependency_links.txt
writing entry points to cvelib.egg-info/entry_points.txt
writing requirements to cvelib.egg-info/requires.txt
writing top-level names to cvelib.egg-info/top_level.txt
reading manifest file 'cvelib.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '__pycache__' found anywhere in distribution
warning: no previously-included files matching '*.py[co]' found anywhere in distribution
writing manifest file 'cvelib.egg-info/SOURCES.txt'
creating /tmp/cvelib/.pybuild/cpython3_3.9/build/man
copying man/cve-list.1 -> /tmp/cvelib/.pybuild/cpython3_3.9/build/man
copying man/cve-org-users.1 -> /tmp/cvelib/.pybuild/cpython3_3.9/build/man
copying man/cve-org.1 -> /tmp/cvelib/.pybuild/cpython3_3.9/build/man
copying man/cve-ping.1 -> /tmp/cvelib/.pybuild/cpython3_3.9/build/man
copying man/cve-publish.1 -> /tmp/cvelib/.pybuild/cpython3_3.9/build/man
copying man/cve-quota.1 -> /tmp/cvelib/.pybuild/cpython3_3.9/build/man
copying man/cve-reject.1 -> /tmp/cvelib/.pybuild/cpython3_3.9/build/man
copying man/cve-reserve.1 -> /tmp/cvelib/.pybuild/cpython3_3.9/build/man
copying man/cve-show.1 -> /tmp/cvelib/.pybuild/cpython3_3.9/build/man
copying man/cve-undo-reject.1 -> /tmp/cvelib/.pybuild/cpython3_3.9/build/man
copying man/cve-user-create.1 -> /tmp/cvelib/.pybuild/cpython3_3.9/build/man
copying man/cve-user-reset-key.1 -> /tmp/cvelib/.pybuild/cpython3_3.9/build/man
copying man/cve-user-update.1 -> /tmp/cvelib/.pybuild/cpython3_3.9/build/man
copying man/cve-user.1 -> /tmp/cvelib/.pybuild/cpython3_3.9/build/man
copying man/cve.1 -> /tmp/cvelib/.pybuild/cpython3_3.9/build/man
copying cvelib/schemas/CVE_JSON_5.0_bundled_5.0.0.json -> /tmp/cvelib/.pybuild/cpython3_3.9/build/cvelib/schemas
copying cvelib/schemas/README.md -> /tmp/cvelib/.pybuild/cpython3_3.9/build/cvelib/schemas
copying cvelib/schemas/adp_container_5.0.0.json -> /tmp/cvelib/.pybuild/cpython3_3.9/build/cvelib/schemas
copying cvelib/schemas/published_cna_container_5.0.0.json -> /tmp/cvelib/.pybuild/cpython3_3.9/build/cvelib/schemas
copying cvelib/schemas/rejected_cna_container_5.0.0.json -> /tmp/cvelib/.pybuild/cpython3_3.9/build/cvelib/schemas
creating /tmp/cvelib/.pybuild/cpython3_3.9/build/tests/data
copying tests/data/CVEv5_advanced-example.json -> /tmp/cvelib/.pybuild/cpython3_3.9/build/tests/data
copying tests/data/CVEv5_basic-example.json -> /tmp/cvelib/.pybuild/cpython3_3.9/build/tests/data
copying tests/data/README.md -> /tmp/cvelib/.pybuild/cpython3_3.9/build/tests/data
root@ba1358ae4029:/tmp/cvelib# ls
CHANGELOG.md Dockerfile LICENSE MANIFEST.in README.md SECURITY.md cvelib cvelib.egg-info man pyproject.toml setup.py tests tox.ini
root@ba1358ae4029:/tmp/cvelib# tree .pybuild/
.pybuild/
|-- cpython3_3.10
| `-- build
| |-- cvelib
| | |-- __init__.py
| | |-- cli.py
| | |-- cve_api.py
| | `-- schemas
| | |-- CVE_JSON_5.0_bundled_5.0.0.json
| | |-- README.md
| | |-- adp_container_5.0.0.json
| | |-- extract_container_schemas.py
| | |-- published_cna_container_5.0.0.json
| | `-- rejected_cna_container_5.0.0.json
| |-- man
| | |-- cve-list.1
| | |-- cve-org-users.1
| | |-- cve-org.1
| | |-- cve-ping.1
| | |-- cve-publish.1
| | |-- cve-quota.1
| | |-- cve-reject.1
| | |-- cve-reserve.1
| | |-- cve-show.1
| | |-- cve-undo-reject.1
| | |-- cve-user-create.1
| | |-- cve-user-reset-key.1
| | |-- cve-user-update.1
| | |-- cve-user.1
| | `-- cve.1
| `-- tests
| |-- data
| | |-- CVEv5_advanced-example.json
| | |-- CVEv5_basic-example.json
| | `-- README.md
| |-- test_cli.py
| `-- test_cve_api.py
|-- cpython3_3.11
| `-- build
| |-- cvelib
| | |-- __init__.py
| | |-- cli.py
| | |-- cve_api.py
| | `-- schemas
| | |-- CVE_JSON_5.0_bundled_5.0.0.json
| | |-- README.md
| | |-- adp_container_5.0.0.json
| | |-- extract_container_schemas.py
| | |-- published_cna_container_5.0.0.json
| | `-- rejected_cna_container_5.0.0.json
| |-- man
| | |-- cve-list.1
| | |-- cve-org-users.1
| | |-- cve-org.1
| | |-- cve-ping.1
| | |-- cve-publish.1
| | |-- cve-quota.1
| | |-- cve-reject.1
| | |-- cve-reserve.1
| | |-- cve-show.1
| | |-- cve-undo-reject.1
| | |-- cve-user-create.1
| | |-- cve-user-reset-key.1
| | |-- cve-user-update.1
| | |-- cve-user.1
| | `-- cve.1
| `-- tests
| |-- data
| | |-- CVEv5_advanced-example.json
| | |-- CVEv5_basic-example.json
| | `-- README.md
| |-- test_cli.py
| `-- test_cve_api.py
`-- cpython3_3.9
`-- build
|-- cvelib
| |-- __init__.py
| |-- cli.py
| |-- cve_api.py
| `-- schemas
| |-- CVE_JSON_5.0_bundled_5.0.0.json
| |-- README.md
| |-- adp_container_5.0.0.json
| |-- extract_container_schemas.py
| |-- published_cna_container_5.0.0.json
| `-- rejected_cna_container_5.0.0.json
|-- man
| |-- cve-list.1
| |-- cve-org-users.1
| |-- cve-org.1
| |-- cve-ping.1
| |-- cve-publish.1
| |-- cve-quota.1
| |-- cve-reject.1
| |-- cve-reserve.1
| |-- cve-show.1
| |-- cve-undo-reject.1
| |-- cve-user-create.1
| |-- cve-user-reset-key.1
| |-- cve-user-update.1
| |-- cve-user.1
| `-- cve.1
`-- tests
|-- data
| |-- CVEv5_advanced-example.json
| |-- CVEv5_basic-example.json
| `-- README.md
|-- test_cli.py
`-- test_cve_api.py
21 directories, 87 files
root@ba1358ae4029:/tmp/cvelib# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
Note that there is no duplicate build directories and only the files that should be included in each package are.
@carnil Where does the pybuild command run exactly?
@carnil I'm going to close this since I can't reproduce it. Feel free to re-open or file a new issue if you find that this is indeed a bug in cvelib. Thanks!
@carnil I'm going to close this since I can't reproduce it. Feel free to re-open or file a new issue if you find that this is indeed a bug in cvelib. Thanks!
@mprpic yes, apologies for the lack of reply. I will need to re-investigate that on my end and will do as you say in case I can find it would be a problem on cvelib side. Thanks for your development on cvelib!