`--skip-existing` gives misleading feedback

[Mr. Lance E Sloan «UMich»]

Is there an existing issue for this?

• I have searched the existing issues (open and closed), and could not find an existing issue

What keywords did you use to search existing issues?

skip

What operating system are you using?

macOS

If you selected 'Other', describe your Operating System here

No response

What version of Python are you running?

$ python --version
Python 3.11.8

How did you install twine? Did you use your operating system's package manager or pip or something else?

$ python3 -m pip install --upgrade twine

What version of twine do you have installed (include the complete output)

$ twine --version
twine version 5.0.0 (importlib-metadata: 7.0.2, keyring: 24.3.1, pkginfo: 1.10.0, requests: 2.31.0, requests-toolbelt: 1.0.0,
urllib3: 2.2.1)

Which package repository are you using?

test.pypi.org

Please describe the issue that you are experiencing

When I run twine upload with the --skip-existing flag, it says it skipped existing files and warnings, but not errors, are given. However, it also shows the colorful progress bar, which appears to indicate that it actually DID upload the skipped files.

Please list the steps required to reproduce this behaviour

1.  $ python3 -m twine upload --repository testpypi --skip-existing dist/*
    Uploading mobyfubarbbq-0.0.1.post1-py3-none-any.whl
    100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 22.7/22.7 kB • 00:00 • 21.7 MB/s
    WARNING  Skipping mobyfubarbbq-0.0.1.post1-py3-none-any.whl because it appears to already exist                                  
    Uploading mobyfubarbbq-0.0.1rc2-py3-none-any.whl
    100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 22.6/22.6 kB • 00:00 • 42.0 MB/s
    Uploading mobyfubarbbq-0.0.1.post1.tar.gz
    100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 36.7/36.7 kB • 00:00 • 67.2 MB/s
    WARNING  Skipping mobyfubarbbq-0.0.1.post1.tar.gz because it appears to already exist                                            
    Uploading mobyfubarbbq-0.0.1rc2.tar.gz
    100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 36.7/36.7 kB • 00:00 • 82.9 MB/s

Anything else you'd like to mention?

I expect it to only show the WARNING lines without the Uploading and progress bar lines for mobyfubarbbq-0.0.1.post1.tar.gz and mobyfubarbbq-0.0.1.post1-py3-none-any.whl.

[Ian Stapleton Cordasco]

We can only determine if something exists reliably if we attempt to upload it. That's why there's a progress bar. If we hide that and only show when successful that could work but I don't believe that provides any value to the user then. Additionally, I don't believe we are able to retroactively hide it but maybe the underlying library has improved since last I looked

[Dustin Ingram]

We can only determine if something exists reliably if we attempt to upload it.

Is this true? package_is_uploaded uses the JSON API to determine if the file already exists:

twine/twine/repository.py

Lines 202 to 232 in 67e87ef

 def package_is_uploaded( 
     self, package: package_file.PackageFile, bypass_cache: bool = False 
 ) -> bool: 
     # NOTE(sigmavirus24): Not all indices are PyPI and pypi.io doesn't 
     # have a similar interface for finding the package versions. 
     if not self.url.startswith((LEGACY_PYPI, WAREHOUSE, OLD_WAREHOUSE)): 
         return False 
  
     safe_name = package.safe_name 
     releases = None 
  
     if not bypass_cache: 
         releases = self._releases_json_data.get(safe_name) 
  
     if releases is None: 
         url = f"{LEGACY_PYPI}pypi/{safe_name}/json" 
         headers = {"Accept": "application/json"} 
         response = self.session.get(url, headers=headers) 
         if response.status_code == 200: 
             releases = response.json()["releases"] 
         else: 
             releases = {} 
         self._releases_json_data[safe_name] = releases 
  
     packages = releases.get(package.metadata.version, []) 
  
     for uploaded_package in packages: 
         if uploaded_package["filename"] == package.basefilename: 
             return True 
  
     return False 

and this happens before upload:

twine/twine/commands/upload.py

Lines 165 to 169 in 67e87ef

 if upload_settings.skip_existing and repository.package_is_uploaded(package): 
     logger.warning(skip_message) 
     continue 
  
 resp = repository.upload(package) 

My read is that we'd only need to upload if a file doesn't appear in the JSON response for a project, and that this would only fail to upload if the file once existed but had been deleted.

[Ian Stapleton Cordasco]

With third party package indices there isn't a JSON API