LookupError: setuptools-scm was unable to detect version if [tool.setuptools_scm] is in pyproject.toml
befeleme opened this issue · comments
project: https://github.com/ionelmc/python-lazy-object-proxy/
download the source code, extract the tar archive and enter the folder: https://github.com/ionelmc/python-lazy-object-proxy/archive/refs/tags/v1.10.0.tar.gz
In pyproject.toml there is empty table [tool.setuptools_scm] defined.
In setup.py you can find the use_scm_version keyword argument with fallback_version explicitly defined.
setup(
name='lazy-object-proxy',
use_scm_version={
'local_scheme': 'dirty-tag',
'write_to': 'src/lazy_object_proxy/_version.py',
'fallback_version': '1.10.0',
}, ...
Running SETUPTOOLS_SCM_DEBUG=1 python setup.py --version ends up with a LookupError:
DEBUG setuptools_scm.overrides dist name: lazy-object-proxy
DEBUG setuptools_scm.config check absolute root=. relative_to=pyproject.toml
DEBUG setuptools_scm.config file pyproject.toml
DEBUG setuptools_scm.entrypoints version_from_ep setuptools_scm.parse_scm in /home/ksurma/Downloads/python-lazy-object-proxy-1.10.0
DEBUG setuptools_scm.discover looking for ep setuptools_scm.parse_scm in /home/ksurma/Downloads/python-lazy-object-proxy-1.10.0
DEBUG setuptools_scm.entrypoints version_from_ep setuptools_scm.parse_scm_fallback in .
DEBUG setuptools_scm.discover looking for ep setuptools_scm.parse_scm_fallback in .
DEBUG setuptools_scm.discover found ep EntryPoint(name='pyproject.toml', value='setuptools_scm.fallbacks:fallback_version', group='setuptools_scm.parse_scm_fallback') in .
DEBUG setuptools_scm.entrypoints EntryPoint(name='pyproject.toml', value='setuptools_scm.fallbacks:fallback_version', group='setuptools_scm.parse_scm_fallback') found None
DEBUG setuptools_scm.discover found ep EntryPoint(name='setup.py', value='setuptools_scm.fallbacks:fallback_version', group='setuptools_scm.parse_scm_fallback') in .
DEBUG setuptools_scm.entrypoints EntryPoint(name='setup.py', value='setuptools_scm.fallbacks:fallback_version', group='setuptools_scm.parse_scm_fallback') found None
DEBUG setuptools_scm.config check absolute root=. relative_to=pyproject.toml
DEBUG setuptools_scm.config file pyproject.toml
Traceback (most recent call last):
File "/home/ksurma/Downloads/python-lazy-object-proxy-1.10.0/setup.py", line 76, in <module>
setup(
File "/usr/lib/python3.12/site-packages/setuptools/__init__.py", line 107, in setup
return distutils.core.setup(**attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 147, in setup
_setup_distribution = dist = klass(attrs)
^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/setuptools/dist.py", line 496, in __init__
_Distribution.__init__(
File "/usr/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 283, in __init__
self.finalize_options()
File "/usr/lib/python3.12/site-packages/setuptools/dist.py", line 935, in finalize_options
ep(self)
File "/home/ksurma/Downloads/python-lazy-object-proxy-1.10.0/.eggs/setuptools_scm-8.0.4-py3.12.egg/setuptools_scm/_integration/setuptools.py", line 121, in infer_version
_assign_version(dist, config)
File "/home/ksurma/Downloads/python-lazy-object-proxy-1.10.0/.eggs/setuptools_scm-8.0.4-py3.12.egg/setuptools_scm/_integration/setuptools.py", line 56, in _assign_version
_version_missing(config)
File "/home/ksurma/Downloads/python-lazy-object-proxy-1.10.0/.eggs/setuptools_scm-8.0.4-py3.12.egg/setuptools_scm/_get_version_impl.py", line 112, in _version_missing
raise LookupError(
LookupError: setuptools-scm was unable to detect version for /home/ksurma/Downloads/python-lazy-object-proxy-1.10.0.
Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.
I comment out the [tool.setuptools_scm] table and run SETUPTOOLS_SCM_DEBUG=1 python setup.py --version again.
No error. Version is correctly set. Please note the found EntryPoint name set to pyproject.toml while in fact all this data is read from setup.py:
DEBUG setuptools_scm.overrides dist name: lazy-object-proxy
DEBUG setuptools_scm.config check absolute root=. relative_to=pyproject.toml
DEBUG setuptools_scm.config file pyproject.toml
DEBUG setuptools_scm.entrypoints version_from_ep setuptools_scm.parse_scm in /home/ksurma/Downloads/python-lazy-object-proxy-1.10.0
DEBUG setuptools_scm.discover looking for ep setuptools_scm.parse_scm in /home/ksurma/Downloads/python-lazy-object-proxy-1.10.0
DEBUG setuptools_scm.entrypoints version_from_ep setuptools_scm.parse_scm_fallback in .
DEBUG setuptools_scm.discover looking for ep setuptools_scm.parse_scm_fallback in .
DEBUG setuptools_scm.discover found ep EntryPoint(name='pyproject.toml', value='setuptools_scm.fallbacks:fallback_version', group='setuptools_scm.parse_scm_fallback') in .
DEBUG setuptools_scm.fallbacks FALLBACK 1.10.0
INFO setuptools_scm.version version 1.10.0 -> 1.10.0
DEBUG setuptools_scm.entrypoints EntryPoint(name='pyproject.toml', value='setuptools_scm.fallbacks:fallback_version', group='setuptools_scm.parse_scm_fallback') found <ScmVersion 1.10.0 dist=0 node=None dirty=False branch=None>
DEBUG setuptools_scm.version scm version <ScmVersion 1.10.0 dist=0 node=None dirty=False branch=None>
DEBUG setuptools_scm.version config Configuration(relative_to='pyproject.toml', root='.', version_scheme='guess-next-dev', local_scheme='dirty-tag', tag_regex=re.compile('^(?:[\\w-]+-)?(?P<version>[vV]?\\d+(?:\\.\\d+){0,2}[^\\+]*)(?:\\+.*)?$'), parentdir_prefix_version=None, fallback_version='1.10.0', fallback_root='.', write_to='src/lazy_object_proxy/_version.py', write_to_template=None, version_file=None, version_file_template=None, parse=None, git_describe_command=None, dist_name='lazy-object-proxy', version_cls=<class 'packaging.version.Version'>, search_parent_directories=False, parent=PosixPath('.'))
DEBUG setuptools_scm.dump_version dump 1.10.0 into src/lazy_object_proxy/_version.py
1.10.0
We came across this issue while debugging a weird behavior of two Fedora's build tools: copr, where the build consistently fails and locally in mock with exactly the same environment, where no such issue arises, the version is read correctly. Could it be that the mere order in which files are read creates such discrepancies?
For the record, when we see a build failure in our build system:
DEBUG setuptools_scm.config file pyproject.toml
DEBUG setuptools_scm.entrypoints version_from_ep setuptools_scm.parse_scm in /builddir/build/BUILD/python-lazy-object-proxy-1.10.0
DEBUG setuptools_scm.discover looking for ep setuptools_scm.parse_scm in /builddir/build/BUILD/python-lazy-object-proxy-1.10.0
DEBUG setuptools_scm.entrypoints version_from_ep setuptools_scm.parse_scm_fallback in .
DEBUG setuptools_scm.discover looking for ep setuptools_scm.parse_scm_fallback in .
DEBUG setuptools_scm.discover found ep EntryPoint(name='pyproject.toml', value='setuptools_scm.fallbacks:fallback_version', group='setuptools_scm.parse_scm_fallback') in .
DEBUG setuptools_scm.entrypoints EntryPoint(name='pyproject.toml', value='setuptools_scm.fallbacks:fallback_version', group='setuptools_scm.parse_scm_fallback') found None
DEBUG setuptools_scm.discover found ep EntryPoint(name='setup.py', value='setuptools_scm.fallbacks:fallback_version', group='setuptools_scm.parse_scm_fallback') in .
DEBUG setuptools_scm.entrypoints EntryPoint(name='setup.py', value='setuptools_scm.fallbacks:fallback_version', group='setuptools_scm.parse_scm_fallback') found None
DEBUG setuptools_scm.config check absolute root=. relative_to=pyproject.toml
DEBUG setuptools_scm.config file pyproject.toml
While when it actually works in a different one:
DEBUG setuptools_scm.config file pyproject.toml
DEBUG setuptools_scm.entrypoints version_from_ep setuptools_scm.parse_scm in /builddir/build/BUILD/python-lazy-object-proxy-1.10.0
DEBUG setuptools_scm.discover looking for ep setuptools_scm.parse_scm in /builddir/build/BUILD/python-lazy-object-proxy-1.10.0
DEBUG setuptools_scm.entrypoints version_from_ep setuptools_scm.parse_scm_fallback in .
DEBUG setuptools_scm.discover looking for ep setuptools_scm.parse_scm_fallback in .
DEBUG setuptools_scm.discover found ep EntryPoint(name='pyproject.toml', value='setuptools_scm.fallbacks:fallback_version', group='setuptools_scm.parse_scm_fallback') in .
DEBUG setuptools_scm.fallbacks FALLBACK 1.10.0
INFO setuptools_scm.version version 1.10.0 -> 1.10.0
DEBUG setuptools_scm.entrypoints EntryPoint(name='pyproject.toml', value='setuptools_scm.fallbacks:fallback_version', group='setuptools_scm.parse_scm_fallback') found <ScmVersion 1.10.0 dist=0 node=None dirty=False branch=None>
The sources should be identical, so I suspect it's the order of files on the filesystem or similar thing we cannot affect.
what versions of setuptools and setuptools_scm are involved
the empty table should be removed upstream
i also recommend setting the pretend version var explicitly
This is setuptools_scm 8.0.4 with setuptools 68.2.2 -- I can test with a newer one...
I was gonna say that setuptools-68.2.2 blows up and setuptools-69.0.3 works correctly, but no, it's just flaky.
I reproduced with Python 3.12.1 and:
Package Version
----------------- -------
packaging 23.2
pip 23.2.1
setuptools 69.0.3
setuptools-scm 8.0.4
typing_extensions 4.9.0
the empty table should be removed upstream
I don't know why they have that. It being empty makes me feel uncomfortable as well. However, at least it should error explicitly or behave consistently if that is the problem.
i also recommend setting the pretend version var explicitly
We can definitively do that when we build the package, but nevertheless, if fallback_version is supposed to work, I guess we should figure out what's happening here, no?
Ah wait, it never should hit fallbacks for GitHub tarballs
Use pypi sdists
Ah wait, it never should hit fallbacks for GitHub tarballs
Why not? It says "fallback". I am confused.
Use pypi sdists
To clarify: we have a workaround available, we are not trying to sort out how to do this, we are trying to figure out what's going on here.
The git archival filename of the project is incorrect, thus it's not found
So it's incorrect repo setup
What does "git archival filename of the project" mean? This happens on unpackaged tarball. If I rename the directory to lazy-object-proxy-1.10.0 it still happens.
The error is in the git repo
That Tag will never produce a viable tarball
I am sorry. I have no idea what you mean.
There is a fallback version set. Sometimes, it is respected, sometimes it is not. I have no idea yet why. If I delete .git-archival.txt I still have the same error. If I clone the repo and delete .git, I have the same error.
What is the meaning of fallback_version, if not the fallback version?
For the record, the error does indeed tell me that what I do is wrong: "Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work."
However, things I don't understand:
- Why does this work sometimes (no sure when yet)?
- Why does this work when we delete pyproject.toml?
- Why does it not fallback to the fallback version?
This is the smallest reproducer I can come up with. Has no git involved whatsoever.
I only have 2 files:
$ cat pyproject.toml
[tool.setuptools_scm]
$ cat setup.py
from setuptools import setup
setup(
name='xxx',
use_scm_version={
'local_scheme': 'dirty-tag',
'write_to': '_version.py',
'fallback_version': '1.10.0',
},
setup_requires=['setuptools_scm'],
)
I create a venv and install latest setuptools and setuptools_scm into it:
$ python3.12 -m venv venv
$ venv/bin/pip install setuptools_scm setuptools
...
Successfully installed packaging-23.2 setuptools-69.0.3 setuptools_scm-8.0.4 typing-extensions-4.9.0
It works:
$ venv/bin/python setup.py --version
...
1.10.0
Now let me uninstall and install setuptools again. This should make no difference:
$ venv/bin/pip uninstall setuptools
...
Successfully uninstalled setuptools-69.0.3
$ venv/bin/pip install setuptools
...
Successfully installed setuptools-69.0.3
Yet suddenly, it no longer works:
$ rm _version.py # optional step, does not change the outcome
$ venv/bin/python setup.py --version
...
Traceback (most recent call last):
File ".../scmscm/setup.py", line 3, in <module>
setup(
File ".../scmscm/venv/lib64/python3.12/site-packages/setuptools/__init__.py", line 103, in setup
return distutils.core.setup(**attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../scmscm/venv/lib64/python3.12/site-packages/setuptools/_distutils/core.py", line 147, in setup
_setup_distribution = dist = klass(attrs)
^^^^^^^^^^^^
File ".../scmscm/venv/lib64/python3.12/site-packages/setuptools/dist.py", line 303, in __init__
_Distribution.__init__(self, dist_attrs)
File ".../scmscm/venv/lib64/python3.12/site-packages/setuptools/_distutils/dist.py", line 283, in __init__
self.finalize_options()
File ".../scmscm/venv/lib64/python3.12/site-packages/setuptools/dist.py", line 654, in finalize_options
ep(self)
File ".../scmscm/venv/lib64/python3.12/site-packages/setuptools_scm/_integration/setuptools.py", line 121, in infer_version
_assign_version(dist, config)
File ".../scmscm/venv/lib64/python3.12/site-packages/setuptools_scm/_integration/setuptools.py", line 56, in _assign_version
_version_missing(config)
File ".../scmscm/venv/lib64/python3.12/site-packages/setuptools_scm/_get_version_impl.py", line 112, in _version_missing
raise LookupError(
LookupError: setuptools-scm was unable to detect version for .../scmscm.
Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.
For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj
A script:
cat > pyproject.toml << EOF
[tool.setuptools_scm]
EOF
cat > setup.py << EOF
from setuptools import setup
setup(
name='xxx',
use_scm_version={
'local_scheme': 'dirty-tag',
'write_to': '_version.py',
'fallback_version': '1.10.0',
},
setup_requires=['setuptools_scm'],
)
EOF
python3.12 -m venv venv
venv/bin/pip install -Uq setuptools_scm setuptools
venv/bin/python setup.py --version # works
venv/bin/pip uninstall -qy setuptools
venv/bin/pip install -q setuptools
venv/bin/python setup.py --version # breaksThat particular error is indeed unexpected
@hroncok thanks for that minimal repro case - I've been investigating a similar problem with the python-trx-python package in Debian, where a _version.py file is unpredictably created-or-not at package build-time -- it has a (non-empty) [tool.setuptools_scm] config table in pyproject.toml.
I believe the unpredictable behaviour is likely to be from the same cause, and that it relates to the order that the version_keyword and infer_version functions in setuptools_scm are invoked during package build.
Findings (so far)
Both the version_keyword and infer_version functions are declared as entrypoints in entry_points.txt file of setuptools_scm:
In the case of version_keyword, the function is placed in the distutils.setup_keywords group, handled by _finalize_setup_keywords in setuptools -- and therefore ultimately becomes part of the same finalize_distribution_options group where infer_version is found. This occurs indirectly, via the entry_points.txt file of setuptools, separate to the entrypoints file of setuptools_scm.
The finalize_distribution_options entrypoint group is a documented distribution-customization feature of setuptools.
Although the finalize_distribution_options group does allow functions to declare an order attribute on their callables to declare a runtime evaluation order, the default ordering is undefined (all items are defaulted to zero) and lacks a tiebreaker comparison.
Issue #1022 seems to relate to this area too.
Observations / possible solutions
What I find locally following installation of setuptools from recent tagged source versions (v69.2.0 for example) is that without changing any of the library code, I can get your minimal test case to pass/fail reliably by simply adjusting the version number in setuptools:setup.cfg and reinstalling it. That doesn't make much sense except to indicate a possible source of nondeterminism that does not relate to the code within setuptools itself, yet that can somehow be affected by (re)installations of the package. Ordering of reads from the filesystem, file timestamps, file paths and cache locations spring to mind as possible explanations, but I haven't confirmed the specific cause in this case yet.
I do think that the nondeterminism could be mitigated somewhere around the finalize_options function by creating a reliable tiebreaker for sort-ordering of the entrypoint functions, or allocating fixed order values for them, but I think it would be better to determine what the origin of the ordering difference is before patching over it.