pipupgrade stuck on Checking... excessive resource usage

[Sam Greadly]

What's the bug all about?

pipupgrade is stuck on "Checking...".

It seems to need a lot of resources (Mem/CPU), which is not feasible on simpler hardware like the Raspberry Pi Zero WH I'm on. Can't seem to handle its needs :/

Linux pizerowh 5.10.17+ #1421 Thu May 27 13:58:02 BST 2021 armv6l GNU/Linux

What command led to this issue?

$ pipupgrade --check

Logs

Please paste the logs using pipupgrade --verbose

"$ pipupgrade --check --verbose
2021-06-13 15:29:49,228 | INFO | Environment: {'version': '1.9.0', 'python_version': '3.7.3', 'os': 'Linux-5.10.17+-armv6l-with-debian-10.9', 'config': {'path': {'BASE': '/home/pi/.local/lib/python3.7/site-packages/pipupgrade', 'DATA': '/home/pi/.local/lib/python3.7/site-packages/pipupgrade/data', 'CACHE': '/home/pi/.config/pipupgrade', 'JOBS': '/home/pi/.local/lib/python3.7/site-packages/pipupgrade/jobs'}}, 'settings': {'settings': {'version': '1.9.0', 'cache_timeout': '86400', 'jobs': '1'}}}
2021-06-13 15:29:49,232 | INFO | Arguments Passed: {'args': (), 'kwargs': {'ARGUMENTS': <class 'inspect._empty'>, 'packages': [], 'ignore': None, 'pip_path': None, 'yes': 0, 'check': True, 'upgrade_type': ['minor', 'patch'], 'latest': 0, 'format': 'table', 'all': 0, 'pip': 0, 'self': False, 'requirements': None, 'pipfile': None, 'interactive': 0, 'project': None, 'git_username': None, 'git_email': None, 'pull_request': False, 'github_access_token': None, 'github_reponame': None, 'github_username': None, 'target_branch': 'master', 'jobs': 4, 'user': 0, 'no_included_requirements': 0, 'no_cache': 0, 'output': None, 'ignore_error': 0, 'force': 0, 'no_color': 0, 'verbose': True}, 'a': <pipupgrade.to_params.<locals>.O object at 0xb672def0>}
Checking...
2021-06-13 15:29:49,306 | INFO | `pip` executables found: ['/home/pi/.local/bin/pip', '/home/pi/.local/bin/pip3', '/usr/bin/pip2']
2021-06-13 15:29:49,309 | INFO | Using 4 jobs...
2021-06-13 15:29:49,623 | INFO | Fetching installed packages for /home/pi/.local/bin/pip...
2021-06-13 15:29:49,633 | INFO | Fetching installed packages for /home/pi/.local/bin/pip3...
2021-06-13 15:29:49,641 | INFO | Fetching installed packages for /usr/bin/pip2...
2021-06-13 15:29:49,664 | INFO | Executing command: /home/pi/.local/bin/pip list --outdated --format json
2021-06-13 15:29:49,669 | INFO | Executing command: /home/pi/.local/bin/pip3 list --outdated --format json
2021-06-13 15:29:49,673 | INFO | Executing command: /usr/bin/pip2 list --outdated --format json"

[Sam Greadly]

Hmm, I'm starting to think this issue is with Python itself, not your script. Although pip-check did manage to update the packages without similar behaviour..

[Jay Sheen]

same, win10, ran pipupgrade, new laptop so not a resources issue, stuck on checking, "pip3 list --outdated" ran fine and instantly.

[Achilles Rasquinha]

Did you try checking #30?

[Achilles Rasquinha]

Going to close, if you believe this is different from #30 feel free to repoen.

[Sam Greadly]

Hi @achillesrasquinha

I need to reopen this. Unfortunately deleting pip's cache didn't work for me.

pipupgrade is still hammering the RPi to the point where it's not progressing past "Checking..."

Why does it need to fork/run that many instances at the same time? Can it not consolidate, or run a few at a time, or sequentially?

[Achilles Rasquinha]

Reopening

[Achilles Rasquinha]

@sgreadly Did you try decreasing the number of jobs it uses.

[Sam Greadly]

Thanks, didn't notice that option. Yes it does help if i set to -j 1. CPU is still 100% solid, but it does go through pip and pip3 at around ~10-15 minutes for each.

I need to fix why it's stopping at pip2 tomorrow then will give it another test.

'''
2021-10-05 16:30:41,767 | INFO | Fetching installed packages for /usr/bin/pip2...
2021-10-05 16:30:41,780 | INFO | Executing command: /usr/bin/pip2 list --outdated --format json

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/home/pi/.local/lib/python3.7/site-packages/pipupgrade/commands/helper.py", line 321, in get_registry_from_pip
format = "json", pip_exec = pip_path, output = True)
File "/home/pi/.local/lib/python3.7/site-packages/pipupgrade/_pip.py", line 75, in call
output = popen(*params, output = output, raise_err = raise_err)
File "/home/pi/.local/lib/python3.7/site-packages/pipupgrade/util/system.py", line 92, in popen
raise PopenError(code, command)
pipupgrade.exception.PopenError: Command '/usr/bin/pip2 list --outdated --format json' returned non-zero exit status 2.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/pi/.local/lib/python3.7/site-packages/pipupgrade/commands/init.py", line 78, in command
return _command(**ARGUMENTS)
File "/home/pi/.local/lib/python3.7/site-packages/pipupgrade/commands/init.py", line 233, in _command
pip_path
File "/usr/lib/python3.7/multiprocessing/pool.py", line 748, in next
raise value
pipupgrade.exception.PopenError: Command '/usr/bin/pip2 list --outdated --format json' returned non-zero exit status 2.

An error occured while performing the above command. This could be an issue with
"pipupgrade". Kindly post an issue at https://github.com/achillesrasquinha/pipupgrade/issues
'''

[Sam Greadly]

Putting pip2 aside as I don't use it, yea managed to it to work with 1 job and just on pip3. Took ~8 minutes.

$ pipupgrade -Vj 1 --pip-path /home/pi/.local/bin/pip3

Thanks! I guess you can close this off.

[Achilles Rasquinha]

@sgreadly You're welcome. pipupgrade tends to do a lot of parallel processing for multiple pip versions within the system simultaneously, it's best to configure according to your needs.