Directory not empty: 'urllib3' when upgrading pip
K-Yo opened this issue · comments
Description
When trying to upgrade pip from 22.0.4 to 22.2.2 on debian, I get the following error during the upgrade process
ERROR: Could not install packages due to an OSError: [Errno 39] Directory not empty: 'urllib3'
afterwards pip binary (/usr/local/bin/pip) does not exist anymore
Another user online seems to have faced the same issue on alpine (for the same pip versions): https://stackoverflow.com/questions/73084494/pip-upgrading-within-the-python-alpine-image-error-could-not-install-packages
Expected behavior
Upgrade does not fail and I can use pip afterwards
pip version
22.0.4
Python version
3.10.5
OS
Debian
How to Reproduce
Using docker (for example):
FROM demisto/python3-deb:3.10.5.31797
RUN pip install --upgrade pipOutput
root@c93e6f957460:/# pip install --no-cache-dir --upgrade pip
Requirement already satisfied: pip in /usr/local/lib/python3.10/site-packages (22.0.4)
Collecting pip
Downloading pip-22.2.2-py3-none-any.whl (2.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 185.6 kB/s eta 0:00:00
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 22.0.4
Uninstalling pip-22.0.4:
ERROR: Could not install packages due to an OSError: [Errno 39] Directory not empty: 'urllib3'
root@c93e6f957460:/# pip
bash: /usr/local/bin/pip: No such file or directoryCode of Conduct
- I agree to follow the PSF Code of Conduct.
The Dockerfile works for me
$ cat Dockerfile
FROM demisto/python3-deb:3.10.5.31797
RUN pip install --upgrade pip
$ docker build .
[+] Building 42.7s (6/6) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 110B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/demisto/python3-deb:3.10.5.31797 3.1s
=> [1/2] FROM docker.io/demisto/python3-deb:3.10.5.31797@sha256:7608af4311f53a6b3df2d43f9b344728cfe91195a7ac39a192dbff56344 24.5s
=> => resolve docker.io/demisto/python3-deb:3.10.5.31797@sha256:7608af4311f53a6b3df2d43f9b344728cfe91195a7ac39a192dbff563445 0.0s
=> => sha256:b41d2e1d193740ed01d296c0bf6a6aeb72501da60451735f8422318ae9e5401b 11.77MB / 11.77MB 7.2s
=> => sha256:7608af4311f53a6b3df2d43f9b344728cfe91195a7ac39a192dbff563445e062 2.84kB / 2.84kB 0.0s
=> => sha256:218a5ed3ee2aabb99331c3455f5d2352823aa0ca40a1b3f7537bd20bb3f91761 10.64kB / 10.64kB 0.0s
=> => sha256:e2be974225eda2dba71b5e519b9cef7d7958fa2bd12448b10eaecb4b95000eeb 1.08MB / 1.08MB 1.4s
=> => sha256:b85a868b505ffd0342a37e6a3b1c49f7c71878afe569a807e6238ef08252fcb7 31.38MB / 31.38MB 22.4s
=> => sha256:f6edef4f72a413275f1493dee2ce8413be5a8223a378294b6e77a4c47a455922 233B / 233B 1.7s
=> => sha256:3968cb080aabd142a7d5a2576e5f3282083fe194b2266e852603383136ca19b4 3.17MB / 3.17MB 6.1s
=> => sha256:7b423f706e3394f5bb44aeecd2c2ffa545cdb6820f11b267addaa10d3cf92204 283B / 283B 6.6s
=> => sha256:00ebe814447aedb9d1a94f0e2f975723110f62c72c90d7b3c05f778febd2a963 684.26kB / 684.26kB 9.6s
=> => sha256:4bf0234f78dd54edb2151d6c2f0c9d7546be6f4daf84487a4efc6c8c7e52bb81 2.75MB / 2.75MB 9.9s
=> => sha256:81b45aaa05cb11f0ae6b7b6eb93161f0b99e5602935877ed117c86c4c54345ad 1.51MB / 1.51MB 12.8s
=> => sha256:f4785a894e2689709d3f26996a20dd678ec8d06f9900aa2327dc4c87b2b34fd0 329B / 329B 10.4s
=> => sha256:f030dbb1c43b64a5bf4d0a1eccfa25ee07b02142171a174c635c1d285f40a332 7.54MB / 7.54MB 16.6s
=> => sha256:0257331ce78c3484f657eb9162eaf12f90d126c46f34fed281bd93d83dd68b9c 5.24kB / 5.24kB 13.3s
=> => extracting sha256:b85a868b505ffd0342a37e6a3b1c49f7c71878afe569a807e6238ef08252fcb7 0.8s
=> => extracting sha256:e2be974225eda2dba71b5e519b9cef7d7958fa2bd12448b10eaecb4b95000eeb 0.1s
=> => extracting sha256:b41d2e1d193740ed01d296c0bf6a6aeb72501da60451735f8422318ae9e5401b 0.3s
=> => extracting sha256:f6edef4f72a413275f1493dee2ce8413be5a8223a378294b6e77a4c47a455922 0.0s
=> => extracting sha256:3968cb080aabd142a7d5a2576e5f3282083fe194b2266e852603383136ca19b4 0.1s
=> => extracting sha256:7b423f706e3394f5bb44aeecd2c2ffa545cdb6820f11b267addaa10d3cf92204 0.0s
=> => extracting sha256:00ebe814447aedb9d1a94f0e2f975723110f62c72c90d7b3c05f778febd2a963 0.0s
=> => extracting sha256:4bf0234f78dd54edb2151d6c2f0c9d7546be6f4daf84487a4efc6c8c7e52bb81 0.1s
=> => extracting sha256:81b45aaa05cb11f0ae6b7b6eb93161f0b99e5602935877ed117c86c4c54345ad 0.1s
=> => extracting sha256:f4785a894e2689709d3f26996a20dd678ec8d06f9900aa2327dc4c87b2b34fd0 0.0s
=> => extracting sha256:f030dbb1c43b64a5bf4d0a1eccfa25ee07b02142171a174c635c1d285f40a332 0.3s
=> => extracting sha256:0257331ce78c3484f657eb9162eaf12f90d126c46f34fed281bd93d83dd68b9c 0.0s
=> [2/2] RUN pip install --upgrade pip 14.9s
=> exporting to image 0.1s
=> => exporting layers 0.1s
=> => writing image sha256:1fb2a3e312b19e2750f81a0abba77f735dbb227a586918efc662372275b2ec83 0.0s
$ docker run $(docker images -q | head -n1) pip --version
pip 22.2.2 from /usr/local/lib/python3.10/site-packages/pip (python 3.10)
@uranusjr came here because I have a bunch of GHA builds failing with similar issues. I have created a reproducer at https://github.com/abn/reproducer-pip-11368
See https://github.com/abn/reproducer-pip-11368/runs/8042452418?check_suite_focus=true#step:3:71 for the specific failure. This has been happening on multiple projects.
The weird part is that the builds work fine locally. Additional point of note here is that in my case the builds are done using buildah not docker.
Verbose Log
Created temporary directory: /tmp/pip-unpack-suo34b4g
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 22.0.4
Uninstalling pip-22.0.4:
Created temporary directory: /tmp/pip-uninstall-nqm3v44c
Removing file or directory /usr/local/bin/pip
Removing file or directory /usr/local/bin/pip3
Removing file or directory /usr/local/bin/pip3.10
Created temporary directory: /usr/local/lib/python3.10/site-packages/~ip-22.0.4.dist-info
Removing file or directory /usr/local/lib/python3.10/site-packages/pip-22.0.4.dist-info/
Created temporary directory: /usr/local/lib/python3.10/site-packages/~ip
ERROR: Could not install packages due to an OSError.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/shutil.py", line 815, in move
os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/usr/local/lib/python3.10/site-packages/pip/' -> '/usr/local/lib/python3.10/site-packages/~ip'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 405, in run
installed = install_given_reqs(
File "/usr/local/lib/python3.10/site-packages/pip/_internal/req/__init__.py", line 68, in install_given_reqs
uninstalled_pathset = requirement.uninstall(auto_confirm=True)
File "/usr/local/lib/python3.10/site-packages/pip/_internal/req/req_install.py", line 638, in uninstall
uninstalled_pathset.remove(auto_confirm, verbose)
File "/usr/local/lib/python3.10/site-packages/pip/_internal/req/req_uninstall.py", line 369, in remove
moved.stash(path)
File "/usr/local/lib/python3.10/site-packages/pip/_internal/req/req_uninstall.py", line 267, in stash
renames(path, new_path)
File "/usr/local/lib/python3.10/site-packages/pip/_internal/utils/misc.py", line 305, in renames
shutil.move(old, new)
File "/usr/local/lib/python3.10/shutil.py", line 833, in move
rmtree(src)
File "/usr/local/lib/python3.10/shutil.py", line 724, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/usr/local/lib/python3.10/shutil.py", line 657, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/usr/local/lib/python3.10/shutil.py", line 663, in _rmtree_safe_fd
onerror(os.rmdir, fullname, sys.exc_info())
File "/usr/local/lib/python3.10/shutil.py", line 661, in _rmtree_safe_fd
os.rmdir(entry.name, dir_fd=topfd)
OSError: [Errno 39] Directory not empty: 'urllib3'
1 location(s) to search for versions of pip:
* https://pypi.org/simple/pip/
Fetching project page and analyzing links: https://pypi.org/simple/pip/
Getting page https://pypi.org/simple/pip/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/pip/" in the cache
Request header has "max_age" as 0, cache bypassed
There was an error checking the latest version of pip
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/pip/_internal/self_outdated_check.py", line 148, in pip_self_version_check
best_candidate = finder.find_best_candidate("pip").best_candidate
File "/usr/local/lib/python3.10/site-packages/pip/_internal/index/package_finder.py", line 868, in find_best_candidate
candidates = self.find_all_candidates(project_name)
File "/usr/local/lib/python3.10/site-packages/pip/_internal/index/package_finder.py", line 809, in find_all_candidates
page_candidates = list(page_candidates_it)
File "/usr/local/lib/python3.10/site-packages/pip/_internal/index/sources.py", line 134, in page_candidates
yield from self._candidates_from_page(self._link)
File "/usr/local/lib/python3.10/site-packages/pip/_internal/index/package_finder.py", line 769, in process_project_url
html_page = self._link_collector.fetch_page(project_url)
File "/usr/local/lib/python3.10/site-packages/pip/_internal/index/collector.py", line 566, in fetch_page
return _get_html_page(location, session=self.session)
File "/usr/local/lib/python3.10/site-packages/pip/_internal/index/collector.py", line 471, in _get_html_page
resp = _get_html_response(url, session=session)
File "/usr/local/lib/python3.10/site-packages/pip/_internal/index/collector.py", line 123, in _get_html_response
resp = session.get(
File "/usr/local/lib/python3.10/site-packages/pip/_vendor/requests/sessions.py", line 542, in get
File "/usr/local/lib/python3.10/site-packages/pip/_internal/network/session.py", line 454, in request
return super().request(method, url, *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/pip/_vendor/requests/sessions.py", line 529, in request
File "/usr/local/lib/python3.10/site-packages/pip/_vendor/requests/sessions.py", line 645, in send
File "/usr/local/lib/python3.10/site-packages/pip/_vendor/cachecontrol/adapter.py", line 57, in send
resp = super(CacheControlAdapter, self).send(request, **kw)
File "/usr/local/lib/python3.10/site-packages/pip/_vendor/requests/adapters.py", line 417, in send
File "/usr/local/lib/python3.10/site-packages/pip/_vendor/requests/adapters.py", line 228, in cert_verify
OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /usr/local/lib/python3.10/site-packages/pip/_vendor/certifi/cacert.pemLooks like issue might have to do with pip running inside rootless containers using fuse-overlays. See containers/fuse-overlayfs#332 (comment).
That issue also references #103.
I can confirm that the workaround specified in the fuse-overlayfs issue resolves this for me. In my case I added the following to the buildah action.
with:
...
extra-args: |
--storage-opt=overlay.mount_program=pip just uses standard Python functions to interact with the filesystem. So does it imply this is a CPython bug?
Welp, I missed the question.
Just checked the build without the mitigation, and it seems it just "works" now (see https://github.com/abn/reproducer-pip-11368/actions/runs/4957985233/jobs/8870312079). No idea if the root cause somehow got fixed somewhere else or if GH just updated their environments.
But for now, I cannot seem to reproduce the issue anymore.
Combining with how the Dockerfile seems to reproduce only on certain environments (the OP’s but not mine), I wonder whether it may actually be a bug in the container layer (GHA also uses containers for their jobs) that is since fixed. In any case I’ll close this for now since there’s nothing further we can do. A new issue can always be opened if a similar issue pops up again.