cannot find python in env when using `pipenv`

[Marc Seitz]

My setup is a repo with a pkgx.yaml file that defines dependencies

dependencies: |
 pipenv.pypa.io@2023.9.1
 python@3.11
 node@20
 npm@10

when I cd into repo, env is set up as expected

...
PKGX_PKGENV=x.org/util-macros=1.20.0 x.org/xau=1.0.11 x.org/xdmcp=1.1.4 x.org/protocol=2023.2.0 x.org/xcb=1.16.0 libpng.org=1.6.40 freetype.org=2.13.2 freedesktop.org/pkg-config=0.29.2 x.org/x11=1.8.7 x.org/exts=1.3.5 curl.se/ca-certs=2023.8.22 gnome.org/libxml2=2.11.4 tea.xyz=0.39.6 zlib.net=1.3.0 sourceware.org/bzip2=1.0.8 openssl.org=1.1.1w sourceware.org/libffi=3.4.4 libexpat.github.io=2.5.0 bytereef.org/mpdecimal=2.5.1 tukaani.org/xz=5.4.4 sqlite.org=3.43.1 gnu.org/readline=8.2.0 invisible-island.net/ncurses=6.4.0 tcl-lang.org=8.6.13 unicode.org=71.1.0 gnu.org/gettext=0.21.1 curl.se=8.3.0 perl.org=5.38.0 pipenv.pypa.io=2023.9.1 python.org=3.11.6 nodejs.org=20.8.0 npmjs.com=10.1.0 git-scm.org=2.42.0 pip.pypa.io=23.2.1
...

Next, I pipenv shell I have a Pipefile already in the repo

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
tinybird-cli = "*"

[dev-packages]

[requires]
python_version = "3.11"

It activates the venv but when I try to update tinybird-cli package it throws an error

(venv) $ pipenv update tinybird-cli
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = '/Users/mfts/.pkgx/pip.pypa.io/v23.2.1'
  program name = '/Users/mfts/.pkgx/pipenv.pypa.io/v2023.9.1/venv/bin/python'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/opt/python.org/v3.11.6/lib/python3.11'
  sys._base_executable = '/Users/mfts/.pkgx/python.org/v3.11.6/bin/python3.11'
  sys.base_prefix = '/opt/python.org/v3.11.6'
  sys.base_exec_prefix = '/opt/python.org/v3.11.6'
  sys.platlibdir = 'lib'
  sys.executable = '/Users/mfts/.pkgx/pipenv.pypa.io/v2023.9.1/venv/bin/python'
  sys.prefix = '/opt/python.org/v3.11.6'
  sys.exec_prefix = '/opt/python.org/v3.11.6'
  sys.path = [
    '/Users/mfts/.pkgx/pip.pypa.io/v23.2.1',
    '/opt/python.org/v3.11.6/lib/python311.zip',
    '/opt/python.org/v3.11.6/lib/python3.11',
    '/opt/python.org/v3.11.6/lib/python3.11/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00000001d7a59ec0 (most recent call first):
  <no Python frame>

The current solution is while the virtual env is active, cd .. && cd <repo> and then it works without the prior error.

(venv) $ pipenv update tinybird-cli
Loading .env environment variables...
Building requirements...
Resolving dependencies...
✔ Success!
Building requirements...
Resolving dependencies...
✔ Success!
Installing dependencies from Pipfile.lock (7ed0ba)...
All dependencies are now up-to-date!

[Max Howell]

I'm not familiar enough with Python so cannot decode its error, what is the error line specicially and what does it mean? Thanks.

[Jacob Heider]

@mfts it would be interesting to see the difference between env output before the pipenv invocation, after, and again after the cd ..; cd -. i assume pipenv is crushing something in the devenv, which is restored when you reenter it.

[Max Howell]

I just tested this with new pip/pipenv and all the other fixes that must have happened since the report and it all works now!