detect-secrets 1.2.0 pre-commit hook errors out with "ModuleNotFoundError: No module named 'requests'"

[Mark Amery]

I think this is a regression; I don't see this error with the 1.1.0 hook and I see that a similar issue (which I can reproduce) was reported in 1.0.0.

If I try to use this hook on my Ubuntu machine...

  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.2.0
    hooks:
      - id: detect-secrets

... then I get this error:

Detect secrets........................................................................................Failed
- hook id: detect-secrets
- exit code: 1

Traceback (most recent call last):
  File "/home/mark/.cache/pre-commit/repovs5glrvx/py_env-python3/bin/detect-secrets-hook", line 5, in <module>
    from detect_secrets.pre_commit_hook import main
  File "/home/mark/.cache/pre-commit/repovs5glrvx/py_env-python3/lib/python3.9/site-packages/detect_secrets/__init__.py", line 1, in <module>
    from .core.secrets_collection import SecretsCollection  # noqa: F401
  File "/home/mark/.cache/pre-commit/repovs5glrvx/py_env-python3/lib/python3.9/site-packages/detect_secrets/core/secrets_collection.py", line 12, in <module>
    from . import scan
  File "/home/mark/.cache/pre-commit/repovs5glrvx/py_env-python3/lib/python3.9/site-packages/detect_secrets/core/scan.py", line 12, in <module>
    from ..filters.allowlist import is_line_allowlisted
  File "/home/mark/.cache/pre-commit/repovs5glrvx/py_env-python3/lib/python3.9/site-packages/detect_secrets/filters/__init__.py", line 2, in <module>
    from . import gibberish     # noqa: F401
  File "/home/mark/.cache/pre-commit/repovs5glrvx/py_env-python3/lib/python3.9/site-packages/detect_secrets/filters/gibberish/__init__.py", line 9, in <module>
    from ...core.plugins import Plugin
  File "/home/mark/.cache/pre-commit/repovs5glrvx/py_env-python3/lib/python3.9/site-packages/detect_secrets/core/plugins/__init__.py", line 1, in <module>
    from . import initialize    # noqa: F401
  File "/home/mark/.cache/pre-commit/repovs5glrvx/py_env-python3/lib/python3.9/site-packages/detect_secrets/core/plugins/initialize.py", line 9, in <module>
    from .util import get_mapping_from_secret_type_to_class
  File "/home/mark/.cache/pre-commit/repovs5glrvx/py_env-python3/lib/python3.9/site-packages/detect_secrets/core/plugins/util.py", line 13, in <module>
    from ...plugins.base import BasePlugin
  File "/home/mark/.cache/pre-commit/repovs5glrvx/py_env-python3/lib/python3.9/site-packages/detect_secrets/plugins/base.py", line 19, in <module>
    import requests
ModuleNotFoundError: No module named 'requests'

Not sure what information about my environment would be useful to aid you debugging this but am happy to answer questions!

[John-Paul Dakran]

@MarkAmeryCurative When you install the detect-secrets package, the install requires the requests package. It should install it when the detect-secrets package is installed. Can you verify that the requests package is installed in your virtual environment?

[Mark Amery]

Hmm. requests is definitely installed in the environment from which I run my Git commands. Originally I was running them with a virualenv enabled, but I tried killing that and just using pyenv without a virtualenv (but again with requests installed), and then finally tried using my system Python without pyenv enabled (again, with requests installed), and still got the same error every time.

I guess it doesn't matter how the Python environment from which I run my Git commands looks anyway, because, based on the traceback above, pre-commit seems to create it's own environment in the ~/.cache/pre-commit directory.

Aha! And it turns out that if I simply delete that folder, everything works again. Interesting.

I've tried to reproduce starting from an empty ~/.cache/pre-commit by e.g. moving between different versions of the hook in my pre-commit-config.yaml, but without success. It seems like the fact that that cache directory was able to get into a broken state indicates a bug somewhere, but without being able to reproduce it, and without having much knowledge of pre-commit's internals, I really have no idea whether to point the finger at detect-secrets, pre-commit itself, or maybe even some other Python-based pre-commit hook I've used other than detect-secrets.

I won't close this myself since perhaps the issue really is in detect-secrets and what I've written above will be enough to make a lightbulb go off in the mind of one of your devs who intimately understands pre-commit... but I will understand if you just decide to close this now that we know it's somehow caused by a bad pre-commit cache but don't have repro instructions.

[John-Paul Dakran]

@MarkAmeryCurative Can you please indicate more information on your configuration. Ubuntu version, pre-commit version etc? Have you tried on a different machine?

[Mark Amery]

Ubuntu 20.04, pre-commit 2.11.1, and haven't tried on a different machine.

[ttibbel]

Hi @MarkAmeryCurative, thanks for the additional information you've provided here. I haven't been able to reproduce this issue using Ubuntu 20.04, pre-commit 2.11.1, and detect-secrets 1.2.0 so I'm thinking this isn't an issue with detect-secrets itself. I'm going to close this issue for now but please feel free to re-open it if you (or anyone else) finds that this is indeed being caused by detect-secrets.