venv with system-site-packages enabled invokes console scripts in the parent version context
Cerebus opened this issue · comments
When a pyenv virtualenv is created with system-site-packages enabled, console scripts installed in the parent pyenv version are invoked using the parent's environment, not the current virtual environment, and cannot load packages installed in the virtualenv.
Reproduction:
Create a venv from a parent version that has packages with console scripts. Here, package pyright has a console script.
> pyenv virtualenv --system-site-packages 3.10.5 foo
> pyenv shell foo
foo > pyenv which pyright
/Users/tmiller/.pyenv/versions/3.10.5/bin/pyright
We pip install a package into the venv.
foo > pip install kopf
Looking in indexes: https://artifacts.mitre.org/artifactory/api/pypi/python/simple
Collecting kopf
Downloading https://artifacts.mitre.org/artifactory/api/pypi/python/packages/packages/80/eb/e82f3e249db2ecd06aeb42eb4209ea295a00a98b9277bff13901e99ddeda/kopf-1.35.6-py3-none-any.whl (205 kB)
[...]
When invoked, pyright parses code and imports, but can't find the package in the venv.
foo > pyright ~/projects/foo
No configuration file found.
No pyproject.toml file found.
stubPath /Users/tmiller/typings is not a valid directory.
Assuming Python platform Darwin
Searching for source files
Found 1 source file
pyright 1.1.269
/Users/tmiller/projects/foo/bar.py
/Users/tmiller/projects/foo/bar.py:3:8 - error: Import "kopf" could not be resolved (reportMissingImports)
1 error, 0 warnings, 0 informations
Completed in 0.596sec
When the console script is invoked as a module entry point, the import is resolved because python is executed in the venv.
foo > python -m pyright ~/projects/foo/
No configuration file found.
No pyproject.toml file found.
stubPath /Users/tmiller/typings is not a valid directory.
Assuming Python platform Darwin
Searching for source files
Found 1 source file
pyright 1.1.269
/Users/tmiller/projects/foo/bar.py
/Users/tmiller/projects/foo/bar.py:6:2 - error: Argument of type "(namespace: Unknown, name: Unknown, resource: Unknown, **_: Unknown) -> dict[tuple[Unknown, Unknown], Unknown]" cannot be assigned to parameter of type "IndexingFn"
Type "(namespace: Unknown, name: Unknown, resource: Unknown, **_: Unknown) -> dict[tuple[Unknown, Unknown], Unknown]" cannot be assigned to type "IndexingFn"
Function accepts too many positional parameters; expected 3 but received 15 (reportGeneralTypeIssues)
1 error, 0 warnings, 0 informations
Completed in 0.82sec
Prerequisite
- Make sure your problem is not listed in the common build problems.
- Make sure no duplicated issue has already been reported in the pyenv-virtualenv issues. You should look in closed issues, too.
There are a couple of close issues, but nothing really seemed to match.
- Make sure you are not asking us to help solving your specific issue.
- Make sure your problem is not derived from packaging (e.g. Homebrew).
- Make sure your problem is not derived from other plugins.
Description
- Platform information (e.g. Ubuntu Linux 20.04):
- OS architecture (e.g. amd64):
Darwin yadda 20.6.0 Darwin Kernel Version 20.6.0: Tue Jun 21 20:50:28 PDT 2022; root:xnu-7195.141.32~1/RELEASE_X86_64 x86_64
- pyenv version:
pyenv 2.3.1-20-g572a8bcf
- pyenv-virtualenv version:
pyenv-virtualenv 1.1.5 (python -m venv)
- Python version:
All.
- virtualenv version (if installed):
N/A
- Please attach the debug log of a faulty Pyenv invocation as a gist
https://gist.github.com/Cerebus/bf0a32edd99d6d8c00a29401628771f8
To clarify the use case: I put things like language servers in site packages because they're used with every project. Project dependencies go in the virtualenv; this isolates the developer's environment from the project environment.
Yes, the workaround is just install developer's dependencies into the venv, but that duplicates things and (depending on the dependency manager) can pollute the environment of collaborators with things they don't need. It's cleaner this way, and with proper pinning/locking, overlaps between site-packages and venv packages aren't a problem.
To further clarify: I see a distinction between developer dependencies and development dependencies. A developer dependency is something a developer uses to be productive and consistent across all projects, but where toolchain may vary from developer to developer. Language servers, linters, and code analysis tools are good examples of these.
In contrast, a development dependency is something required by a project but is not part of the release. Build and test frameworks are the typical examples.
So when I say "developer dependency" above, that's what I mean. I don't want to install a language server as a development dependency, because (a) it's not needed to build, test, or run the code; and (b) you might use a different one.
Virtualenvs with system-site-packages allows this to work, but pyenv is giving me fits. This actually is working on my personal Linux workstation, but poking around in there I see that there's something wonky on that install; pyenv which is giving me the scripts in $HOME/.local rather than the parent venv. I think this may have to do with how I migrated that environment to pyenv, whereas this macOS box was a cleanroom install. Once I figure out what's going on, it may suggest a workaround or a fix.
Try this. It worked for me
That works for imports, not for console scripts.