pip freeze dumping system package list with shell=zsh in a user's ~/.vexrc
yurimalheiros opened this issue · comments
When I use "shell=bash" in my .vexrc file, and type the following commands:
$ vex myenv
$ pip freeze
everything is ok, but if I use "shell=zsh", and type the same commands, I get the packages list installed in my system, the same list if I do not use the vex/virtualenvs.
I use vex with zsh every day, so I know it is not accurate to say that it does not work well with zsh. pip freeze also works without any issue. I would suggest that this means something relating to which pip or some environment variable specific to your user account or machine. Perhaps something is going on in ~/.zshrc?
Thank you for the reply. I suspected that it couldn't be a general problem.
What kind of thing in the .zshrc may cause this problem?
Sorry. I have tried a few times to replicate this and haven't been able to do it. That's not your fault.
If you want to help with troubleshooting this issue you are experiencing, please do this.
Make a brand new virtualenv with
vex -m newenv
then with each of (empty .vexrc, .vexrc with shell=bash, .vexrc with shell=zsh):
vex newenv pip freeze
vex newenv which pip
vex newenv bash -c 'echo $PATH'
vex newenv
which pip
echo PATH
exit
and report the outputs here (after making sure they don't contain any sensitive info).
If you can't help with this then I will close the issue and you can open a new one when you have more detail for me to work with.
Thanks for your help finding bugs in vex.
Hi.
I did the tests.
My default shell is zsh.
empty .vexrc
$ vex newenv pip freeze
Traceback (most recent call last):
File "/usr/local/bin/vex", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/site-packages/vex/main.py", line 161, in main
returncode = _main(os.environ, argv)
File "/usr/local/lib/python2.7/site-packages/vex/main.py", line 145, in _main
env = get_environ(environ, vexrc['env'], ve_path)
File "/usr/local/lib/python2.7/site-packages/vex/run.py", line 47, in get_environ
segments.remove(current_ve_bin)
ValueError: list.remove(x): x not in list
$ vex newenv which pip
Traceback (most recent call last):
File "/usr/local/bin/vex", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/site-packages/vex/main.py", line 161, in main
returncode = _main(os.environ, argv)
File "/usr/local/lib/python2.7/site-packages/vex/main.py", line 145, in _main
env = get_environ(environ, vexrc['env'], ve_path)
File "/usr/local/lib/python2.7/site-packages/vex/run.py", line 47, in get_environ
segments.remove(current_ve_bin)
ValueError: list.remove(x): x not in list
$ vex newenv bash -c 'echo $PATH'
Traceback (most recent call last):
File "/usr/local/bin/vex", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/site-packages/vex/main.py", line 161, in main
returncode = _main(os.environ, argv)
File "/usr/local/lib/python2.7/site-packages/vex/main.py", line 145, in _main
env = get_environ(environ, vexrc['env'], ve_path)
File "/usr/local/lib/python2.7/site-packages/vex/run.py", line 47, in get_environ
segments.remove(current_ve_bin)
ValueError: list.remove(x): x not in list
$ vex newenv
Traceback (most recent call last):
File "/usr/local/bin/vex", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/site-packages/vex/main.py", line 161, in main
returncode = _main(os.environ, argv)
File "/usr/local/lib/python2.7/site-packages/vex/main.py", line 145, in _main
env = get_environ(environ, vexrc['env'], ve_path)
File "/usr/local/lib/python2.7/site-packages/vex/run.py", line 47, in get_environ
segments.remove(current_ve_bin)
ValueError: list.remove(x): x not in list
$ which pip
/usr/local/bin/pip
$ echo $PATH
/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/texlive/2012/bin/universal-darwin:/Users/yurimalheiros/.rvm/bin
.vexrc with shell=bash
$ vex newenv pip freeze
wsgiref==0.1.2
$ vex newenv which pip
/Users/yurimalheiros/.virtualenvs/newenv/bin/pip
$ vex newenv bash -c 'echo $PATH'
/Users/yurimalheiros/.virtualenvs/newenv/bin:/Users/yurimalheiros/.rvm/gems/ruby-2.0.0-p247/bin:/Users/yurimalheiros/.rvm/gems/ruby-2.0.0-p247@global/bin:/Users/yurimalheiros/.rvm/rubies/ruby-2.0.0-p247/bin:/Users/yurimalheiros/.rvm/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/texlive/2012/bin/universal-darwin
$ vex newenv
(it enters in bash shell)
$ which pip
/Users/yurimalheiros/.virtualenvs/newenv/bin/pip
$ echo $PATH
/Users/yurimalheiros/.virtualenvs/newenv/bin:/Users/yurimalheiros/.rvm/gems/ruby-2.0.0-p247/bin:/Users/yurimalheiros/.rvm/gems/ruby-2.0.0-p247@global/bin:/Users/yurimalheiros/.rvm/rubies/ruby-2.0.0-p247/bin:/Users/yurimalheiros/.rvm/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/texlive/2012/bin/universal-darwin:/Users/yurimalheiros/.rvm/bin
.vexrc with shell=zsh
$ vex newenv pip freeze
wsgiref==0.1.2
$ vex newenv which pip
/Users/yurimalheiros/.virtualenvs/newenv/bin/pip
$ vex newenv bash -c 'echo $PATH'
/Users/yurimalheiros/.virtualenvs/newenv/bin:/Users/yurimalheiros/.rvm/gems/ruby-2.0.0-p247/bin:/Users/yurimalheiros/.rvm/gems/ruby-2.0.0-p247@global/bin:/Users/yurimalheiros/.rvm/rubies/ruby-2.0.0-p247/bin:/Users/yurimalheiros/.rvm/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/texlive/2012/bin/universal-darwin
$ vex newenv
(it enters in zsh)
$ which pip
/usr/local/bin/pip
$ echo $PATH
/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/texlive/2012/bin/universal-darwin:/Users/yurimalheiros/.rvm/bin
It is interesting, in zsh, if I use "vex newenv pip freeze" it works well, but if I use "vex newenv", and then "pip freeze", I get the package list of my system.
I hope this help you.
Peculiar: segments.remove(current_ve_bin) is failing in the first few tests because somehow you have set the VIRTUAL_ENV environment variable, BUT its bin/ is not in PATH! VIRTUAL_ENV normally gets set by activating a virtualenv (using source env/bin/activate, virtualenvwrapper's workon myenv, or vex). which should also update PATH. So vex is trying to remove env/bin from PATH because it must be there, since we are in a virtualenv, but env/bin was not put in PATH by whatever tool that set VIRTUAL_ENV (and therefore should have set PATH).
If PATH is not being set correctly (which is indicated by that segments.remove call failing!) then 'vex foo pip' will find the wrong pip executable - the system-wide one instead of the one from the activated virtualenv - and you will get the wrong output from pip.
Maybe you can figure out what is setting VIRTUAL_ENV at the beginning? Because at least one bug is that whatever it is isn't setting PATH at the beginning.
vex edits the PATH for the process it's spawning at the line which says 'segments.insert(0, ve_bin)'
I think the first errors occurred, because I ran "vex -m newenv", and this command create and activate the virtualenv, so, when I ran the following commands in the "empty .vexrc" case, it got a VIRTUAL_ENV already set.
I tried again. I created the virtualenv with "vex -m newenv", leaved the env, and ran the commands. Now, everything in the "empty .vexrc" case is equal to ".vexrc with shell=zsh" case.
Is this ok? My default shell is zsh, so, when I don't set anything in .vexrc, the shell used is zsh. Right?
It's good, thanks, what you've done is very helpful.
Short version: would you try temporarily renaming .zshrc (so zsh won't source it) and seeing what happens with vex newenv? You can rename it back soon/later, this just helps us figure out if it's something happening in .zshrc when zsh is run by vex.
Longer version:
The first errors aren't because of vex -m, they're because some process other than vex messed up the environment before vex ran: it promised it would properly activate a virtualenv by setting VIRTUAL_ENV, but then it failed to complete the promise because PATH was not set up correctly.
One possibility is that something DID correctly set both VIRTUAL_ENV and PATH... but then something else set PATH incorrectly, so that the wrong pip is being found and so on.
An additional reason to suspect that possibility is your last test with 'vex newenv'. I know that vex is updating PATH to put the virtualenv bin at the start of PATH. However, not only is the resulting PATH starting with '/usr/local/bin' - it doesn't include any virtualenv bin directory at all. Although you just ran vex newenv. I don't think there is any code path in which vex would fail to update path. Even if some other checks failed and it prepended only an empty string, we would see a telling leading colon in PATH. However, it's very plausible that something run in .zshrc is setting PATH to some other value (e.g. not just appending to the PATH it sees).
So you might look in .zshrc or other .z* (files only sourced by zsh - since things look fine for bash) for things which set PATH.
empty .vexrc should be roughly the same as shell=zsh, yes.
To be precise, what happens is that when vex is run with one argument (the virtualenv name) and no command, it looks at $SHELL. On my system, for example, 'echo $SHELL' gives '/bin/zsh'. So strictly speaking it is identical to 'shell=/bin/zsh' in vexrc. IIRC, 'shell=zsh' should rely on PATH to find the right executable to run. However, this is probably a red herring insofar as something else is munging PATH.
I got a different result now.
$ vex newenv zsh -c 'echo $PATH'
/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/usr/texbin:/Users/yurimalheiros/.virtualenvs/newenv/bin:/Users/yurimalheiros/.rvm/gems/ruby-2.0.0-p247/bin:/Users/yurimalheiros/.rvm/gems/ruby-2.0.0-p247@global/bin:/Users/yurimalheiros/.rvm/rubies/ruby-2.0.0-p247/bin:/Users/yurimalheiros/.rvm/bin
$ vex newenv which pip
/Users/yurimalheiros/.virtualenvs/newenv/bin/pip
and
$ vex newenv
$ echo $PATH
/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/usr/texbin:/Users/yurimalheiros/.virtualenvs/newenv/bin:/Users/yurimalheiros/.rvm/gems/ruby-2.0.0-p247/bin:/Users/yurimalheiros/.rvm/gems/ruby-2.0.0-p247@global/bin:/Users/yurimalheiros/.rvm/rubies/ruby-2.0.0-p247/bin:/Users/yurimalheiros/.rvm/bin
$ which pip
/usr/local/bin/pip
I got the same results for "echo $PATH", but the "which pip" and, consequently, "pip freeze" were still different.
That is without a .zshrc?
yes
It's hard for me to know what is going on in your configs. You could try shell=/bin/zsh instead of shell=zsh but I doubt it will make a difference because I assume you have just one zsh and they'd behave the same anyway. I wouldn't bother.
What I need to know is what, in zsh's configs, is changing PATH after vex starts zsh. Almost certainly vex is handing over a correct PATH to the new zsh, but then something sourced by zsh is setting PATH incorrectly.
Personally I would be grepping .* for PATH and setting debug flags on my shell to find out where PATH is changing, but I can't give you very detailed guidance on how to hunt through your files.
Is this possibly an interaction with rvm? Is some rvm stuff being sourced at zsh startup or something?
I guess if this persists I can have you email me your zsh configs and I can try to use them to replicate the issue you're seeing, but maybe first see if you can track down where PATH is getting munged yourself (after vex started running the zsh process, e.g. during zsh startup/sourcing zsh configs).
try
shell=zsh -f
this should try zsh without any configs being loaded
The result of "which pip" is still different with "shell=zsh -f" configuration.
I'm going to investigate this problem more. I really appreciated your help until now, thank you :)
This bug seems caused probably by some configuration of my computer.
If I discovered something new, I will comment here, or I will contact you using other way if you want to close this issue.
Hmm that is very interesting, I will try some other ideas of wrecking this another time. Thanks for all your patient help!
My wife pulled an old laptop out of the closet and saw a similar issue (though the shells were reversed, bash was failing instead of zsh).
After I got momentarily confused because PATH was being set both in ~/.profile (executed by login shells) and in ~/.bashrc, it turned out that /usr/local/bin was being prepended to the beginning of PATH in ~/.bashrc (which is of course executed for the start of non-login shells like what vex will typically start, and this will override vex's PATH modification). Since /usr/local/bin was ahead of the virtualenv's bin in PATH, and itself contained a pip, the wrong pip was executing.
I corrected ~/.bashrc so that /usr/local/bin wasn't prepended to PATH at every new shell start - and I once again can't reproduce the problem.
So I guess it is worth triple-checking all the places you are setting or exporting PATH in your dotfiles, to make sure that one of them is not overriding PATH shortly after shell startup.
I guess you could post your configs somewhere (.bashrc, .zshrc, .profile, etc.) and I could look at them, but otherwise I don't think I can replicate this. Would like to help, so feel free to reopen if you get more information for me to look at or if you come to a diagnosis after more investigating