Lektor cached venv is broken on macOS with Python installed via Homebrew
hatarist opened this issue · comments
I'm using the latest version (Lektor-3.4.0b9.dev7+gbfce91a.d20230827 as of now).
python3 is installed via homebrew (python@3.11
) and is symlinked:
/usr/local/bin/python3
-> /usr/local/Cellar/python@3.11/3.11.5/bin/python3
-> ... -> /usr/local/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/bin/python3.11
I have created a venv using python3 -m venv env
, everything worked fine in that venv until I installed a plugin (lektor plugins add lektor-index-pages
). After that, trying to run the lektor server
started raising an exception:
...
File "/Users/igor/.virtualenvs/lektor/lib/python3.11/site-packages/lektor/packages.py", line 125, in create
EnvBuilder(upgrade_deps=upgrade_deps, **options).create(self.path)
...
subprocess.CalledProcessError: Command '['/Users/igor/Library/Caches/Lektor/venvs/d63a0228b40929c7189d2c99d18e960b/bin/python3.11', '-m', 'ensurepip', '--upgrade', '--default-pip']' died with <Signals.SIGABRT: 6>.
When I tried to run python3 using that path, I saw this error:
dyld[34820]: Library not loaded: @loader_path/../../../../Python.framework/Versions/3.11/Python
Referenced from: <430E089A-35B0-314B-9FD1-1E60F9CE5756> /Users/igor/Library/Caches/Lektor/venvs/d63a0228b40929c7189d2c99d18e960b/bin/python3.11
Reason: tried: '/Users/igor/Library/Caches/Lektor/venvs/d63a0228b40929c7189d2c99d18e960b/bin/../../../../Python.framework/Versions/3.11/Python' (no such file), '/Library/Frameworks/Python.framework/Versions/3.11/Python' (no such file), '/System/Library/Frameworks/Python.framework/Versions/3.11/Python' (no such file, not in dyld cache)
[1] 34820 abort
So, looks like the homebrew version of Python was built with relative paths in mind and when EnvBuilder copies that binary to another path, things go south.
I have successfully fixed that for myself by editing the EnvBuilder options in lektor/packages.py:123, adding an symlinks=True
option there:
options: Dict[str, Any] = {"clear": True, "with_pip": with_pip, "symlinks": True}
This has solved the issue for me - now, when Lektor creates a cached venv, it symlinks the used python binary in there and everything seems to work fine.
I've never heard of Lektor until today and I don't feel comfortable with making a pull request since my fix could break something else. Sorry!
Thank you for the report!
At this point, I only vaguely understand how symlinking vs copying might make a difference, but...
It appears the venv CLI (python -m venv
), by default, uses symlinks everywhere except Windows. (Some versions/configurations of Windows do not fully support symlinks, or do not allow their creation without administrator-level access.)
In contrast, venv.EnvBuilder
(as you note) defaults to not using symlinks anywhere.
So, I think an appropriate fix on our side would be just to duplicate the venv CLI's default logic to determine when to use symlinks. (I.e. pass symlinks=os.name != "nt"
to the EnvBuilder
constructor.)
If you're so inclined, feel free to make a PR to do that. Otherwise, I'll work one up in the next few days.
@hatarist This should now be fixed in the master branch. If not, please re-open the issue.
Thank you for the report and the analysis!