Incorrect username when using system keyring CLI

[Pineapple Pokopo]

Describe the bug
I use PDM for Python package management and Google Artifact Registry (GAR) as a private registry and PyPi mirror using the following pyproject.toml config:

[[tool.pdm.source]]
url = "https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple"
verify_ssl = true
name = "pypi"

I'd like to use keyring and keyrings.google-artifactregistry-auth to authenticate to GAR. I can enable this by installing these packages into the PDM environment by running:

pdm self add keyring keyrings.google-artifactregistry-auth

and authentication then works:

unearth.collector: Collecting links from https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/
unearth.auth: Found index url https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
unearth.auth: Found index url https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
unearth.auth: Getting credentials from keyring for url: https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
unearth.auth: Found credentials in keyring for europe-west4-python.pkg.dev
unearth.collector: Fetching HTML page https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/

However, when using a system-installed copy of keyring and keyrings.google-artifactregistry-auth, authentication fails:

unearth.collector: Collecting links from https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/
unearth.auth: Found index url https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
unearth.auth: Found index url https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
unearth.auth: Getting password from keyring CLI for __token__@https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
unearth.auth: Found credentials in keyring for europe-west4-python.pkg.dev
unearth.auth: 401 Error, Credentials not correct for https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/
unearth.collector: Failed to collect links from https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/: Client Error(401): Unauthorized

I believe this is the result of an incorrect user (__token__) being set here:

unearth/src/unearth/auth.py

Lines 82 to 83 in b43f3bf

 if username is None: 
     username = "__token__" 

The keyring module returns the correct user (oauth2accesstoken) however:

Python 3.11.7 (main, Jan  2 2024, 11:13:25) [Clang 15.0.0 (clang-1500.1.0.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import keyring
>>> creds = keyring.get_credential("http://europe-west4.python.pkg.dev", None)
>>> creds.username
'oauth2accesstoken'

To Reproduce
In an environment with keyring and keyrings.google-artifactregistry-auth available globally, run:

python -m unearth --verbose --index-url 'https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/' --trusted-host 'europe-west4-python.pkg.dev' setuptools

Output:

DEBUG: Collecting links from https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/
DEBUG: Found index url https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
DEBUG: Found index url https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
DEBUG: Getting password from keyring CLI for __token__@https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
DEBUG: Found credentials in keyring for europe-west4-python.pkg.dev
WARNING: 401 Error, Credentials not correct for https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/
WARNING: Failed to collect links from https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/: Client Error(401): Unauthorized
No matches are found.

Install keyring and keyrings.google-artifactregistry-auth, .e.g.: pdm add keyring keyrings.google-artifactregistry-auth and run python -m unearth --verbose --index-url 'https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/' --trusted-host 'europe-west4-python.pkg.dev' setuptools again.

Output:

DEBUG: Collecting links from https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/
DEBUG: Found index url https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
DEBUG: Found index url https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
DEBUG: Getting credentials from keyring for url: https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
DEBUG: Found credentials in keyring for europe-west4-python.pkg.dev
DEBUG: Fetching HTML page https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/

Expected behavior

Both keyring as module and as standalone CLI being able to authenticate to GAR.

System (please complete the following information):

• unearth version: 0.14.0
• Python version: 3.11.7
• OS: macOS 13.6.4

Additional context
Related to failing to run PDM as part of Renovate, see renovatebot/renovate#27454, where keyring and keyrings.google-artifactregistry-auth cannot be easily injected into the PDM environment, but only installed globally.

[Frost Ming]

keyring CLI doesn't has the equivalent of keyring.get_credential with non-specified username.

[Pineapple Pokopo]

Right, I see. Do you know of any work around? I tried setting the username explicitly through env var PDM_PYPI_USERNAME, but this also fails:

$ export PDM_PYPI_USERNAME=oauth2accesstoken
$ pdm add setuptools -vv 
Adding packages to default dependencies: setuptools
STATUS: Resolving dependencies
pdm.termui: ======== Start resolving requirements ========
pdm.termui:   setuptools
pdm.termui:   python>=3.11,<3.12
pdm.termui:   Adding requirement setuptools
unearth.collector: Collecting links from https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/
unearth.auth: Getting password from keyring CLI for oauth2accesstoken@pdm-pypi-pypi
unearth.auth: Found index url https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
unearth.auth: Getting password from keyring CLI for oauth2accesstoken@pdm-pypi-pypi
unearth.auth: Found index url https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
unearth.auth: Getting password from keyring CLI for __token__@https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
unearth.auth: Found credentials in keyring for europe-west4-python.pkg.dev
unearth.auth: 401 Error, Credentials not correct for https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/
unearth.collector: Failed to collect links from https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/: Client Error(401): Unauthorized

[Frost Ming]

unearth.auth: Getting password from keyring CLI for oauth2accesstoken@pdm-pypi-pypi
unearth.auth: Found index url https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/

It is actually taking effect, but the url seems to be pdm-pypi-pypi, please check that.

[Pineapple Pokopo]

Yes, I noticed that as well, but I haven't set the url explicitly. It seems to be coming from here: https://github.com/pdm-project/pdm/blob/429df860c822b519a88f71e0d38d2e308ffcfec2/src/pdm/_types.py#L46.

For completeness, what I did:

1. Create an empty directory
2. Runpdm init
3. Add to pyproject.toml:

   [[tool.pdm.source]]
   url = "https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple"
   verify_ssl = true
   name = "pypi"
   

4. Run: export PDM_PYPI_USERNAME=oauth2accesstoken
5. Run: pdm add setuptools -vv

Output:

Adding packages to default dependencies: setuptools
STATUS: Resolving dependencies
pdm.termui: ======== Start resolving requirements ========
pdm.termui:   setuptools
pdm.termui:   python>=3.11,<3.12
pdm.termui:   Adding requirement setuptools
unearth.collector: Collecting links from https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/
unearth.auth: Getting password from keyring CLI for oauth2accesstoken@pdm-pypi-pypi
unearth.auth: Found index url https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
unearth.auth: Getting password from keyring CLI for oauth2accesstoken@pdm-pypi-pypi
unearth.auth: Found index url https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
unearth.auth: Getting password from keyring CLI for __token__@https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/
unearth.auth: Found credentials in keyring for europe-west4-python.pkg.dev
unearth.auth: 401 Error, Credentials not correct for https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/
unearth.collector: Failed to collect links from https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/: Client Error(401): Unauthorized
Traceback (most recent call last):
  File "/Users/<USER>/.asdf/installs/pdm/2.12.3/bin/pdm", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/<USER>/.asdf/installs/pdm/2.12.3/venv/lib/python3.11/site-packages/pdm/core.py", line 288, in main
    return Core().main(args or sys.argv[1:])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/<USER>/.asdf/installs/pdm/2.12.3/venv/lib/python3.11/site-packages/pdm/core.py", line 208, in main
    raise cast(Exception, err).with_traceback(traceback) from None
  File "/Users/<USER>/.asdf/installs/pdm/2.12.3/venv/lib/python3.11/site-packages/pdm/core.py", line 203, in main
    self.handle(project, options)
  File "/Users/<USER>/.asdf/installs/pdm/2.12.3/venv/lib/python3.11/site-packages/pdm/core.py", line 157, in handle
    command.handle(project, options)
  File "/Users/<USER>/.asdf/installs/pdm/2.12.3/venv/lib/python3.11/site-packages/pdm/cli/commands/add.py", line 69, in handle
    self.do_add(
  File "/Users/<USER>/.asdf/installs/pdm/2.12.3/venv/lib/python3.11/site-packages/pdm/cli/commands/add.py", line 159, in do_add
    resolved = do_lock(
               ^^^^^^^^
  File "/Users/<USER>/.asdf/installs/pdm/2.12.3/venv/lib/python3.11/site-packages/pdm/cli/actions.py", line 102, in do_lock
    mapping, dependencies = resolve(
                            ^^^^^^^^
  File "/Users/<USER>/.asdf/installs/pdm/2.12.3/venv/lib/python3.11/site-packages/pdm/resolver/core.py", line 39, in resolve
    result = resolver.resolve(requirements, max_rounds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/<USER>/.asdf/installs/pdm/2.12.3/venv/lib/python3.11/site-packages/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/<USER>/.asdf/installs/pdm/2.12.3/venv/lib/python3.11/site-packages/resolvelib/resolvers.py", line 397, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/Users/<USER>/.asdf/installs/pdm/2.12.3/venv/lib/python3.11/site-packages/resolvelib/resolvers.py", line 173, in _add_to_criteria
    if not criterion.candidates:
  File "/Users/<USER>/.asdf/installs/pdm/2.12.3/venv/lib/python3.11/site-packages/resolvelib/structs.py", line 127, in __bool__
    next(iter(self))
  File "/Users/<USER>/.asdf/installs/pdm/2.12.3/venv/lib/python3.11/site-packages/pdm/resolver/providers.py", line 356, in matches_gen
    yield from super_find()
               ^^^^^^^^^^^^
  File "/Users/<USER>/.asdf/installs/pdm/2.12.3/venv/lib/python3.11/site-packages/pdm/resolver/providers.py", line 232, in matches_gen
    candidates = self._find_candidates(reqs[0])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/<USER>/.asdf/installs/pdm/2.12.3/venv/lib/python3.11/site-packages/pdm/resolver/providers.py", line 202, in _find_candidates
    return self.repository.find_candidates(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/<USER>/.asdf/installs/pdm/2.12.3/venv/lib/python3.11/site-packages/pdm/models/repositories.py", line 173, in find_candidates
    cans = LazySequence(self._find_candidates(requirement, minimal_version=minimal_version))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/<USER>/.asdf/installs/pdm/2.12.3/venv/lib/python3.11/site-packages/pdm/models/repositories.py", line 443, in _find_candidates
    raise CandidateNotFound(
pdm.exceptions.CandidateNotFound: Unable to find candidates for setuptools. There may exist some issues with the package name or network condition.

UPDATE:

Also tried adding the username to the index url directly in pyproject.toml:

[[tool.pdm.source]]
url = "https://[e-mail]/<PROJECT>/<REPOSITORY>/simple/"
verify_ssl = true
name = "pypi"

But this results in a similar error:

$ pdm add setuptools -vv 
Adding packages to default dependencies: setuptools
STATUS: Resolving dependencies
pdm.termui: ======== Start resolving requirements ========
pdm.termui:   setuptools
pdm.termui:   python==3.11.*
pdm.termui:   Adding requirement setuptools
unearth.collector: Collecting links from https://***@europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/
unearth.auth: Found index url https://[e-mail]/<PROJECT>/<REPOSITORY>/simple/
unearth.auth: Found index url https://[e-mail]/<PROJECT>/<REPOSITORY>/simple/
unearth.auth: Getting password from keyring CLI for __token__@https://[e-mail]/<PROJECT>/<REPOSITORY>/simple/
unearth.auth: Found credentials in keyring for europe-west4-python.pkg.dev
unearth.auth: 401 Error, Credentials not correct for https://europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/
unearth.collector: Failed to collect links from https://***@europe-west4-python.pkg.dev/<PROJECT>/<REPOSITORY>/simple/setuptools/: Client Error(401): Unauthorized