`rtx activate` incompatible with direnv

Question

`rtx activate` incompatible with direnv

amoosbr opened this issue 2 years ago · comments

Alexander Moosbrugger commented 2 years ago

I started playing with rtx today to replace my asdf-direnv setup.

When using layout python3 from the direnv stdlib, it used my global rtx python3 version and not the rtx local one. In the shell itself, the venv of direnv wasn't used, as rtx added the configured path sections to the fron of PATH.

Updated: ~~My fix was to move eval "$(rtx activate -s zsh)" after my direnv one in .zshrc~~ The work-around is only working in the folder containing .envrc and .tool-versions, but not in subfolders (see comment below):

eval "$(direnv hook zsh)"
eval "$(rtx activate -s zsh)"

Is this something for a known-issues/documentation section or can the zsh call sequence somehow be influenced?

Thank you for this initial release of rtx. It's a great start.

Reproduce:

~/.tool-versions:

python 3.10.9

.zshrc

eval "$(rtx activate -s zsh)"
eval "$(direnv hook zsh)"

~/dev/tmp/rtx-direnv/.envrc:

layout python3

~/dev/tmp/rtx-direnv/.tool-versions:

python 3.8.10

Steps:

❯ echo $PATH
/Users/user/.local/share/rtx/installs/python/3.10.9/bin:/opt/homebrew/opt/mysql-client/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Users/user/.rd/bin:/opt/homebrew/opt/fzf/bin
❯ cd ~/dev/tmp/rtx-direnv
direnv: error /Users/user/Projects/tmp/rtx-direnv/.envrc is blocked. Run `direnv allow` to approve its content

# after the initial `direnv allow` it will seem to work, as the path is already adjusted by rtx
❯ direnv allow
direnv: loading ~/Projects/tmp/rtx-direnv/.envrc
direnv: export +VIRTUAL_ENV ~PATH
❯ cd ..
direnv: unloading
❯ cd ~/dev/tmp/rtx-direnv
direnv: loading ~/Projects/tmp/rtx-direnv/.envrc
direnv: export +VIRTUAL_ENV ~PATH

# direnv venv path overwritten and venv pointing to wrong python version
❯ echo $PATH
/Users/user/.local/share/rtx/installs/python/3.8.10/bin:/Users/user/Projects/tmp/rtx-direnv/.direnv/python-3.10.9/bin:/opt/homebrew/bin:/Users/user/bin:/Users/user/go/bin:/Users/user/.local/bin:/Users/user/.cargo/bin:/opt/homebrew/opt/mysql-client/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Users/user/.rd/bin:/opt/homebrew/opt/fzf/bin

With updated .zshrc:

eval "$(direnv hook zsh)"
eval "$(rtx activate -s zsh)"

Steps:

# open new shell
❯ cd ~/dev/tmp/rtx-direnv
direnv: loading ~/Projects/tmp/rtx-direnv/.envrc
direnv: export +VIRTUAL_ENV ~PATH
❯ echo $PATH
/Users/user/Projects/tmp/rtx-direnv/.direnv/python-3.8.10/bin:/Users/user/.local/share/rtx/installs/python/3.8.10/bin:/opt/homebrew/bin:/Users/user/bin:/Users/user/go/bin:/Users/user/.local/bin:/Users/user/.cargo/bin:/opt/homebrew/opt/mysql-client/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin

Jeff Dickey · Answer 1 · Sat Jan 28 2023 04:04:23 GMT+0800 (China Standard Time)

I may have to think about this one a bit. direnv and rtx are likely conflicting with each other. They both maintain a list of "diffs" between what the env was before it ran on directory change as well as after. So what is likely happening is rtx says "here is the state before", then direnv makes some changes and rtx later overwrites them to put things back the way they were. If that makes sense.

I am pretty sure this is a problem with both rtx and direnv since if you put them in opposite order then direnv would overwrite what rtx is doing.

I think in the short term I'd be careful to not manage the same environment variable with both (chiefly PATH). I could potentially check DIRENV_DIFF to see what changes it has and at least warn if I detect this situation.

Bojan Delić · Answer 2 · Sat Jan 28 2023 06:58:23 GMT+0800 (China Standard Time)

I just stumbled on the same problem, but with fish.

My flow is to choose Python version via .tool-versions locally and let direnv activate (or create and activate) virtualenv using currently selected version using layout python.

Workaround with switching order does not really work for me, since I have installed direnv via homebrew and it is being activated by fish automatically (loading /opt/homebrew/share/fish/vendor_conf.d/direnv.fish), not in my fish config. Although I guess I could install direnv via https://github.com/asdf-community/asdf-direnv and manually initialise it.

Maybe solution might be Python virtualenv specific, not direnv specific? E.g. instead of detecting DIRENV_DIFF environment variable, maybe rtx could detect if environment variable VIRTUAL_ENV if it is set and skip setting path to Python defined in .tool-versions? Not sure which consequences such change might have on the rest of the system (using rtx about 20 minutes :) ), but one consequence, I presume, would be to keep activated virtual env when switching directories, regardless if direnv activated it or it was done manally. Of course, this would be Python specific solution, not general one, so if there is general solution - even better.

I am happy to provide additional info if needed regarding my setup.

Jeff Dickey · Answer 3 · Sat Jan 28 2023 07:12:52 GMT+0800 (China Standard Time)

@delicb you've said a couple of things that concern me so I want to be very clear:

I did not state a workaround with switching order. I said switching the order likely won't help.
I would not recommend trying to use asdf-direnv with rtx. That's not likely to work well and there isn't any reason to with how fast rtx is.

In terms of direnv generally, the solution appears to skip using rtx activate and instead call rtx from within direnv. This is possible using direnv_load and I'm currently working on some docs and built-in tools to make this process easy. It's not that hard to fix on your own either. I also have been thinking it may be possible for me to detect DIRENV_DIFF and actually work within that field to intelligently not conflict with direnv—but that might be challenging if it even would work.

I don't think any of this has to do with virualenv in particular but I suppose we'll see.

Short term, you can try disabling rtx activate and putting this into envrc:

watch_file .tool-versions
export RTX_MISSING_RUNTIME_BEHAVIOR=warn
direnv_load rtx exec -- direnv dump

That seems to work for me in basic testing.

Bojan Delić · Answer 4 · Sat Jan 28 2023 07:20:59 GMT+0800 (China Standard Time)

@jdxcode Thanks for your response.

@amoosbr mentioned switching order worked for him, that is what I was referring to. I am not planning using asdf-direnv plugin with rtx, I just mentioned it as mental exercise :)

Thanks for the snippet, I will give it a try.

Bojan Delić · Answer 5 · Sun Jan 29 2023 07:56:39 GMT+0800 (China Standard Time)

@jdxcode While approach suggested in README.md for working with direnv now works, it means that rtx is not globally activated, which means that if there is no .envrc file $PATH is not set. This means that executing quick scripts in random directory will not work (even having .envrc in $HOME is not always enough, sometimes some tools are needed in /tmp or in /etc).

Do you have a suggestion for making tools defined in ~/.tool-versions available globally, while for project based development keep direnv working?

Thanks.

Jeff Dickey · Answer 6 · Sun Jan 29 2023 08:11:42 GMT+0800 (China Standard Time)

I've been thinking about this but haven't come up with a great idea so far. Here are some ideas I've considered:

Try using rtx activate again. I doubt it will work, but I have put some changes in that should mitigate conflicts between the two
Support reading/writing DIRENV_DIFF env vars within rtx to prevent direnv from modifying changes. This seems possible but pretty challenging. The current logic is already probably the scariest part of the entire rtx codebase. This would make it a lot worse.
Use source_up in all of your envrc files (though I note what you've said about /tmp and stuff)

However here is what I think would be best but I haven't tried it myself:

First, call eval "$(rtx env -s zsh)" in your ~/.zshrc before direnv so direnv won't overwrite anything we set. I know you mentioned something about having it installed with a conf.d script but just remove that file and put direnv hook fish | source in your config.fish. Note that I said rtx env—not rtx activate. This only runs a single time, it won't keep the environment variables up-to-date.

What that will do is set some global runtimes for you. You can put it into ~/.tool-versions and that will be read even if you're in /tmp or whatever. So now your global runtimes are covered and you just need it to be overridden in project directories.

For project runtimes you can use the rtx direnv integration. You'll still need to always have a envrc file with use rtx inside of it (so far as I can figure out), but you can setup your watch file stuff so that modifying a .tool-versions file in a child directory will cause envrc to reload and fetch new env vars from rtx.

It's not the greatest solution but it should work.

Jeff Dickey · Answer 7 · Sun Jan 29 2023 08:18:47 GMT+0800 (China Standard Time)

To expand on my DIRENV_DIFF idea: the way that direnv (and rtx) works is it tracks what the environment variable state was before direnv set env vars from .envrc. The problem we're having is that env vars that rtx creates after direnv will be overwritten by direnv because it thinks it needs to reset the state back to where it was before.

If I modify that bookkeeping—which is a little hairy because it's like a base64 encoded, compressed json blob, I can trick direnv into thinking that some new environment variables were in place before it so it won't clobber things back. That's relatively straightforward with ad-hoc env vars like JAVA_HOME, but challenging for PATH (which is most of what we care bout).

Jeff Dickey · Answer 8 · Sun Jan 29 2023 08:24:19 GMT+0800 (China Standard Time)

One other idea is that we could use shims. Of course if you've read the README you know I don't like them, but the overhead of calling them with rtx is negligible—probably only 2-3ms.

You can do the following:

$ cat node> ~/shims/bin/node <<EOF
#!/bin/sh
rtx x -- node "$@"
EOF
$ chmod +x ~/shims/bin/node
$ ~/shims/node -v
18.0.0
$ export PATH="$HOME/shims/node:$PATH"

Now in any directory rtx will have that shim point to the correct node runtime.

Jeff Dickey · Answer 9 · Sun Jan 29 2023 08:38:03 GMT+0800 (China Standard Time)

Ok one more crazy idea—this one really is bonkers.

In theory rtx can be named whatever you want. The CLI is a single file and whatever its name is is its name. That means you can name it asdf. If you did mv ~/bin/rtx ~/bin/asdf then you could install asdf shims with rtx because they would be calling asdf exec but that would actually be launching rtx.

However the commands between the 2 are not identical. They might be close enough though. I'd be happy to find out places were there are miscompatibilities and see how we could get them to align, even if we had to use some sort of "compat" mode. We might need this anyways since plugins commonly call asdf.

You could also do a hybrid approach and create an asdf binary that is your own shim. If it is called with asdf exec, then it dispatches to rtx for speed, if it's anything else it dispatches to asdf. You would probably want to set RTX_DATA_HOME=$HOME/.asdf—which may or may not function very well.

Anyways I realize this is probably way too much information but as you can tell I'm still trying to think of how best to handle this situation. If you have some time to experiment with any of these ideas please let me know how it goes. I wish this would've been as simple as me just patching rtx to not conflict with direnv but I think that might be challenging.

I suppose one other idea is for me to reach out to direnv and see if they have any ideas.

Bojan Delić · Answer 10 · Sun Jan 29 2023 08:40:42 GMT+0800 (China Standard Time)

Thanks for the quick reply and explanation.

I did try using rtx activate again, but as you suspected - it did not work - same problem as initially described in this issue.

I trust you when you say that this would complicate things. I am not Rust dev so I haven't looked at source code, but I did read direnv source code couple years ago and I understand potential complications. It is probably not worth implementing unless there is more demand or there are more use cases that required this. If I understand correctly, DIRENV_DIFF contains format that is internal for direnv and not a part of public API and that could change, thus breaking implementation and making it even less worthy of maintaining.

I did set up project to use source_up and source_up_if_exists in my projects and in home directory I created ~/.envrc with use rtx. But this does not work as expected in terms of not respecting local .tool-versions file. I am guessing that .tool-versions is read from the same directory where rtx was invoked. With this setup, version of runtime used is one defined in $HOME/.tool-versions, so I dropped that approach.

Adding rtx env -s fish | source to config.fish does seem to solve the problem. Changing global tool versions is not that frequent for this to be problematic. So far, I like this approach the best.

For using shims - I am not a fan either and I totally understand your logic behind not using them. I would not go down that road unless absolutely necessary. It did cross my mind to use the same approach you mentioned (I even created a small script to generate these shims for all files in ~/.local/share/rtx/installs/*/*/bin, but I dropped it and decided to ask here if there is different suggestion.

Let me use rtx with use rtx in .envrc files and rtx env -s fish | source in config.fish and see if there are any potential problems or nuances.

Thanks again for the response, help and awesome tool :).

Bojan Delić · Answer 11 · Sun Jan 29 2023 08:45:43 GMT+0800 (China Standard Time)

Hah, I was typing during at the same time while you were sending your last comment.

I did not thing of using rtx as a shim layer. But that does make sense, I was even locally patching asdf with https://github.com/danhper/asdf-exec for performance reasons until I switched to rtx, which has similar idea, but at the different level (asdf shims call asdf-exec, which is natively bash script, and this replaces it with Go implementation, so shims still exist, execution is faster (from ~100ms to ~10ms on my machine for starting Python)).

It is definitely interesting idea, I will give it a try one in the next following days and write up findings.

Jeff Dickey · Answer 12 · Sun Jan 29 2023 08:46:37 GMT+0800 (China Standard Time)

So in regards to local .tool-version files, it's not that rtx only looks at ones in the same directory as the envrc file, it will actually look at all the parents of it as well. The problem is that I have no way of knowing where children of the envrc file will be and I have to tell direnv up front which .tool-version files to look out for and cause a change to be triggered. Remember when you use the direnv integration it's direnv that requests new env vars from rtx, we have no good way to tell direnv to expect one.

This is possibly fixable. Perhaps we could have a special shell extension that we just use for communicating with direnv and telling it to trigger a change. Or maybe we could cache recent locations of all of your tool-version files or something.

Jeff Dickey · Answer 13 · Sun Jan 29 2023 08:51:15 GMT+0800 (China Standard Time)

The thing about asdf-direnv and asdf-exec is I really don't think a good solution is to bring some kind of addon to asdf. Now you've got 3 different tools to maintain. I used asdf-direnv for a bit but it did behave very strangely a time or two and I couldn't figure out if the problem was with asdf, direnv, or with asdf-direnv.

Maybe what I need to do is make rtx a complete direnv replacement. It already is capable of loading environment variables in directories like direnv does. I don't think I could ever get to the level of scripting functionality that direnv provides in .envrc files (certainly can't imagine reading them), but for me all I ever do with direnv is export FOO=bar which is already supported.

The hard part about this is we'd need a new file since .tool-versions isn't a good place for environment variables. I have a TOML format but it needs work.

Jeff Dickey · Answer 14 · Sun Jan 29 2023 09:03:54 GMT+0800 (China Standard Time)

Yet another idea (this one more practical): you could do your configuration for rtx in .envrc files as environment variables and not use .tool-version files at all. That way you don't need to worry about maintaining 2 separate files, you can just set export RTX_NODEJS_VERSION=18.2.1 in your .envrc and it will behave as if you had nodejs 18.2.1 in .tool-versions.

Practically everything should work the same, this just gives you 1 instead of 2 files to keep updated.

Jeff Dickey · Answer 15 · Sun Jan 29 2023 09:16:43 GMT+0800 (China Standard Time)

I'm going to keep this task open in the hope that someone might notice it and have a good idea to share.

Alexander Moosbrugger · Answer 16 · Mon Jan 30 2023 16:36:40 GMT+0800 (China Standard Time)

Thanks for all the thoughts you put into solving the rtx/direnv conflict.

Doing some more testing, I can confirm, that my initial work-around with the "correct" zsh hook order is only working in the directory containing .envrc and .tools-version. As soon as I continue to traverse a directory further down in the project, rtx will overwrite the changes of direnv.

❯ ls .envrc .tool-versions
.envrc  .tool-versions

# rtx and direnv updated the path, as expected
❯ echo $PATH
/Users/user/Projects/x/node_modules/.bin:/Users/user/Projects/x/.direnv/python-3.9.12/bin:/Users/user/.local/share/rtx/installs/python/3.9.12/bin:...

# on directory change, direnv changes are overwritten
❯ cd apps
❯ echo $PATH
/Users/user/.local/share/rtx/installs/python/3.9.12/bin:...

I think for me, the way you described above is feasible and I will give it a try. So far looks good.

Unfortunately, I found an issue with GOROOT and RUBYLIB environment variables. As the issue is within direnv, but also when used without direnv, I opened a new issue #47

Jeff Dickey · Answer 17 · Tue Jan 31 2023 04:45:34 GMT+0800 (China Standard Time)

I had an idea here. I was thinking rtx is in a better position in regards to updating PATH because it knows that the only things it adds to PATH are prefixed with ~/.local/share/rtx/installs.

I'm already dropping those paths explicitly every time rtx starts. That alone means that I can not worry about replacing PATH to what it was before the hook is run, just leave everything as it was but remove rtx directories.

This means that rtx wouldn't override anything set by the user or direnv. However there is still the inverse: direnv will overwrite what rtx sets.

For that we will need to go with my original idea of parsing DIRENV_DIFF and taking it to think the new rtx paths were in place before direnv ran. Or perhaps if we simply always ran direnv before rtx that might be enough. Hard to think through exactly how this would behave.

This works for PATH but not if you're setting GOROOT and stuff with direnv and rtx. That feels much less important though.

Jeff Dickey · Answer 18 · Tue Jan 31 2023 12:17:24 GMT+0800 (China Standard Time)

This is probably the most important issue concerning me at the moment. I think the idea I sketched out above should work though. It'll take me a bit to implement it but when I'm able to get around to it I hope rtx will be fully compatible without the use_rtx function.

Alexander Moosbrugger · Answer 19 · Tue Jan 31 2023 17:36:47 GMT+0800 (China Standard Time)

I think, your idea of 'in-place' updates would work in combination with direnv. And I hope, you also don't need to care about DIRENV_DIFF, when rtx and direnv run in the 'right' order.

Based on what I tested with the current rtx integration (not using use rtx), I think, the rtx hook should run before the direnv hook though. At least to use the full capabilities of direnv. And in my mind it makes sense. rtx uses .tool-versions to set the default PATH for a project. Direnv needs this project PATH already set, pointing to the correct project tool binaries, when it runs .envrc. I.e. it uses the PATH content in its standard library functions like layout python3,... and might further modify the PATH.
As long as rtx just does 'in-place' upgrades, when switching to subdirectories, within the same .envrc/.tool-versions ancestor, it should still work (This doesn't work today).
A corner case is, if within a sub-directory a new .envrc file exists. But I think, as long as the rtx hook ran before the initial direnv hook, direnv would reset the PATH to the one seen by the initial direnv hook. And this contained already the correct project PATH set by rtx.

If the rtx hook would run after the direnv hook, direnv would see the PATH binaries pointing to the global .tool-versions binaries, while running the .envrc content, and i.e. functions like layout python3 might use the wrong global python version instead of a project one.

And I think with this order, also the GOROOT settings could work. But it's possible, if direnv modified i.e. GOROOT, that they get overwritten by rtx, when switching to a subdirectory. Perhaps here, you would need to read DIRENV_DIFF? But I agree, that solving the issue with the PATH is more relevant.

I hope, you could follow my thoughts. And can evaluate, if they made sense or not. It's totally possible, I missed something or have a wrong understanding of hooks/rtx/direnv.
And thank you for searching a better solution then the current one with use rtx and not using rtx activate.

Jeff Dickey · Answer 20 · Wed Feb 01 2023 08:24:29 GMT+0800 (China Standard Time)

1.2.9 is out with direnv updates, including patching DIRENV_DIFF. This means you should be able to use rtx activate alongside direnv without use rtx. In my basic testing it seemed to work well except at one point it made my PATH invalid, clearing most of it out and putting " ." at the end. Obviously that's bad, but I couldn't figure out what caused it or make it happen again. I retraced my steps exactly.

For that reason I'm leaving in the guidance in the README to say not to use it, but if you wouldn't mind trying to test it I would really appreciate it. I'm really hoping that bug was just a one-off from development or something.

I realized that it doesn't matter what the order is because you can have directories with envrc files or with tool-versions files so ultimately direnv and rtx are going to run at different times regardless. So they have to be able to execute in either order. (Perhaps not with use rtx though, that's different).

I'm hoping we're at least on the right track for getting direnv and rtx to play nice.

Alexander Moosbrugger · Answer 21 · Wed Feb 01 2023 17:55:52 GMT+0800 (China Standard Time)

I tested this a bit this morning with two sample projects and it really improved a lot.
Correction: ~~And up till now, I didn't run into the issue you mentioned with the cleared out PATH.~~ Ran into the issue with the cleared out PATH as well and added a comment, with the steps to reproduce.

For my use-cases of direnv and rtx the order of the hooks is still relevant though.

Given the .envrc and .tool-versions are in the same directory:

.envrc:

layout python3
PATH_add "foo"

.tool-versions:

python 3.8.10

It is working as expected for me with

.zshrc:

eval "$(direnv hook zsh)"
eval "$(rtx activate --quiet -s zsh)"

Resulting in the PATH:

❯ cd rtx-env-variables-direnv
direnv: loading ~/Projects/tmp/rtx-env-variables-direnv/.envrc
direnv: export +VIRTUAL_ENV ~PATH
❯ echo $PATH
/Users/user/Projects/tmp/rtx-env-variables-direnv/foo:/Users/user/Projects/tmp/rtx-env-variables-direnv/.direnv/python-3.8.10/bin:/Users/user/.local/share/rtx/installs/python/3.8.10/bin:...
❯ python --version
Python 3.8.10
❯ which python
/Users/user/Projects/tmp/rtx-env-variables-direnv/.direnv/python-3.8.10/bin/python
❯ echo $VIRTUAL_ENV
/Users/user/Projects/tmp/rtx-env-variables-direnv/.direnv/python-3.8.10

If I switch the order of the both hooks, the result is:

.zshrc:

eval "$(rtx activate --quiet -s zsh)"
eval "$(direnv hook zsh)"

 cd rtx-env-variables-direnv
direnv: loading ~/Projects/tmp/rtx-env-variables-direnv/.envrc
direnv: export +VIRTUAL_ENV ~PATH

# PATH order not as I would expect
❯ echo $PATH
/Users/user/.local/share/rtx/installs/python/3.8.10/bin:...:/Users/user/Projects/tmp/rtx-env-variables-direnv/foo:/Users/user/Projects/tmp/rtx-env-variables-direnv/.direnv/python-3.10.9/bin:...
❯ python --version
Python 3.8.10
❯ which python
/Users/user/.local/share/rtx/installs/python/3.8.10/bin/python

# VIRTUAL_ENV set by layout python3 used global python version. Not the local one
❯ echo $VIRTUAL_ENV
/Users/user/Projects/tmp/rtx-env-variables-direnv/.direnv/python-3.10.9

As you can see the order of the added paths in PATH differs, but more importantly the content as well.
Direnvs layout python3 function will create a virtual environment based on the Python version found in the PATH, when the direnv hook was called.
With the rtx hook running after direnv, the virtual environment is using the, in my opinion, wrong Python version "3.10.9" from the global PATH. And therefore VIRTUAL_ENV points to the wrong directory.

I only did a limited test of course. So it is possible, there are use cases for both orders. Just wanted to mention it.
With this change I have now a working rtx and direnv combination, that doesn't need any change in .envrc, but still be way faster then the previous asdf and/or asdf-direnv setup. Thank you. 👍🎉

~~For me the issue can now be closed, when you have enough confidence regarding the clear PATH issue you mentioned.~~

Alexander Moosbrugger · Answer 22 · Wed Feb 01 2023 18:52:45 GMT+0800 (China Standard Time)

Unfortunately, I must correct myself. I ran also in the issue with the cleared out PATH.
But was able to reproduce. Unfortunately, I can not try to create a e2e test today. Sorry.

It occurred for me, when directly switching from one project using only direnv .envrc to a project using only rtx .tool-versions.

Steps to reproduce:

Note: Set export RTX_DEBUG=1 once, but didn't see anything in the output, that looked to be relevant.

.zshrc (not sure, if relevant to reproduce):

eval "$(direnv hook zsh)"
eval "$(rtx activate --quiet -s zsh)"

❯ cd ~/dev/tmp/rtx-direnv-break-path-source
direnv: loading ~/Projects/tmp/rtx-direnv-break-path-source/.envrc
direnv: export +FOO
❯ ll
.rw-r--r--@ 17 user staff  1 Feb 11:34 .envrc
❯ cat .envrc
export FOO='foo'
❯ ll ../rtx-direnv-break-path-target
.rw-r--r--@ 14 user staff  1 Feb 11:43 .tool-versions
❯ cat ../rtx-direnv-break-path-target/.tool-versions
python 3.8.10
❯ cd ~/dev/tmp/rtx-direnv-break-path-target
direnv: unloading
__zoxide_hook:2: command not found: zoxide
direnv: error can't find bash: exec: "bash": executable file not found in $PATH
❯ echo $PATH
/Users/user/.local/share/rtx/installs/python/3.8.10/bin:
direnv: error can't find bash: exec: "bash": executable file not found in $PATH

Jeff Dickey · Answer 23 · Wed Feb 01 2023 22:10:22 GMT+0800 (China Standard Time)

you're very good at QA @amoosbr :)

I also was able to reproduce this on a different computer once but again wasn't able to retrace my steps. One thing that would be helpful is if I added RTX_LOG_FILE_LOG_LEVEL as an environment variable so we could dump the trace logs to a file and not have them litter our consoles. I'll see if I can add that this morning. EDIT: #70

Now let me see if I can repro what you've put here.

EDIT: nice! this does reproduce my issue! Nice work!

Jeff Dickey · Answer 24 · Thu Feb 02 2023 03:10:27 GMT+0800 (China Standard Time)

I figured it out. In the case where .envrc did not edit the PATH, rtx would add it but not the entire thing. We should just ignore DIRENV_DIFF if PATH isn't one of the variables listed.

Jeff Dickey · Answer 25 · Thu Feb 02 2023 04:06:24 GMT+0800 (China Standard Time)

@amoosbr let me know if you see any more issues but I think this is resolved.

Bojan Delić · Answer 26 · Thu Feb 02 2023 06:33:49 GMT+0800 (China Standard Time)

@jdxcode I did some testing and current version (1.3.1) produces some strange results for me. I will attach screenshot.

As you can see, I created new folder and added .envrc to it that just sets python version and calls layout python. Python version is not respected, layout python i still using 3.11 to create virtualenv (which is my global python version).

(Don't be confused with location of virtualenv, I like to keep it separate from project itself, so I have direnv configured to create it in ~/.cache/direnv/layouts/<project-name>/python-<version>, but it is visible from the path which version is used and python --version confirms that).

I have also noticed another problem, but I was unable to determine when it happens - sometimes when I cd into a project, direnv changes to the path do not take effect (layout python exists in .envrc, but which python points to $HOME/.local/share/rtx/installs/python). I am not sure what is causing it, but I noticed that touch .envrc usually fixes it.

I am currently using setup with rtx activate -s fish | source in config script and no use rtx lines in .envrc files. Is there anything else I am missing? Is order of activating rtx and direnv important now?
For now I will revert to rtx env -s fish | source + use rtx method.

Jeff Dickey · Answer 27 · Thu Feb 02 2023 07:37:44 GMT+0800 (China Standard Time)

This all actually makes sense to me. You can't just set RTX_PYTHON_VERSION in an envrc and immediately call layout python and expect those versions to be set because rtx hasn't had a chance to run and modify the PATH.

As far as local changes not taking effect, you'll get different behavior based on whether rtx or direnv runs first if you're modifying the same executable like python. If rtx runs last, it'll put its stuff on top, direnv will put its stuff on top and there isn't a way for either tool to control that.

The fixes I've put in place have been around changes to PATH for different paths. Before, direnv would remove rtx's modifications to anything.

In terms of what you should do, I think you can keep using rtx activate. Just in your envrc force direnv to set the path to the python you want:

PATH_add $(rtx where python@3.10)/bin
layout python

I think that will get you the behavior you're looking for without rtx always needing to run before direnv—which isn't always possible. For one, you could have a subdirectory with a .envrc file and no .tool-version file so rtx would have to reason to execute at all.

Alexander Moosbrugger · Answer 28 · Thu Feb 02 2023 16:24:05 GMT+0800 (China Standard Time)

@delicb I saw the similar behavior as you did. See this comment with details. I think, the order of the hooks triggering is still relevant. The rtx hook must run before direnv to get direnv to behave as I'm expecting.

@jdxcode Great news, I also wasn't able to reproduce the cleared out PATH with version 1.3.2 anymore. Thanks for the quick fix. 🎉
For me the issue can now be closed, when you have enough confidence/feedback and updated the README.

Bojan Delić · Answer 29 · Thu Feb 02 2023 20:09:47 GMT+0800 (China Standard Time)

@amoosbr Thanks, you are right, order is relevant. However, for me, order of hooks that works is different:

rtx activate -s fish | source
direnv hook fish | source

Works, while if I set it the other way around, I always get python from $HOME/.local/share/rtx/installs first in path.

Maybe it has something to do with the shell (I see you are using zsh and I am using fish).

@jdxcode thanks for clarification. What i did was to uninstall direnv installed via homebrew and download binary directly, so that I can control the moment hooks are initiated (since order matters).

I have created $HOME/.config/direnv/lib/use_python.sh with following content (based on your suggestion and similar to other tools direnv supports with use function):

use_python() {
    local version=${1:-}
    echo "using python ${version}"
    PATH_add $(rtx where python@${version})/bin
}

So my .envrc now looks like:

use python 3.10
layout python

and as far as I can tell, this are working as expected now.

Thanks for all the time you spent on this issue 🙌

Alexander Moosbrugger · Answer 30 · Thu Feb 02 2023 20:19:01 GMT+0800 (China Standard Time)

@delicb Yeah, maybe the sourcing order and executing order of the hooks differs between the two. I didn't understand exactly, why you still need the use python 3.10 code in your .envrc file. In zsh at least, with the hooks executing in the right order for me just layout python3 is sufficient. Might also be specific to my setup or zsh vs fish.
Glad you also found a working solution.

Bojan Delić · Answer 31 · Thu Feb 02 2023 20:21:03 GMT+0800 (China Standard Time)

@amoosbr how are you controlling which version of python you want to use for project? Via local .tool-versions? I have both 3.10 and 3.11 installed and use different versions for different project, so this is my way to explicitly set right version so that layout python can pick it up. I guess .tool-versions would work as well.

Edit: I just tried using rtx local python@3.10 which creates .tool-versions locally and it seem to work just like my approach with use python. Still not sure which one I prefer better, will decide over time, both seems to be working.

Alexander Moosbrugger · Answer 32 · Thu Feb 02 2023 20:26:11 GMT+0800 (China Standard Time)

Yes. I'm using .tool-versions. Just didn't read your code well enough. But great, that both options exist and are working.

Alexander Moosbrugger · Answer 33 · Thu Feb 02 2023 23:21:59 GMT+0800 (China Standard Time)

Found another issue, that I think is somehow related to DIRENV_DIFF as well. Sorry :-)

Issue is, that when traversing away from a direnv and rtx managed folder to an outside folder not managed by direnv, rtx doesn't clean up the PATH in a correct way. First, I thought this has low priority, as the left-overs were set after the new ones rtx added. But with system versions it is relevant, as in this case suddenly you're not using the expected tool version anymore.

This time, I also created a simple e2e test to recreate the behavior #79.
If you prefer, I can also create a new issue to track this. Just let me know.

Steps to manually reproduce:

❯ cd e2e/direnv/system_version
rtx: nodejs@system
❯ cat .tool-versions
nodejs system
❯ node -v
v19.5.0

# with rtx nodejs@system no nodejs path segment in PATH
❯ echo $PATH
/Users/user/bin:...

❯ cd rtx-direnv-system-version-reset/load-first
rtx: nodejs@17.7.1
direnv: loading ~/rtx/e2e/direnv/system_version/rtx-direnv-system-version-reset/load-first/.envrc
direnv: export +FIRST ~PATH
❯ node -v
v17.7.1
❯ ll
.rw-r--r--@ 96 user staff  2 Feb 15:48 -- .envrc
.rw-r--r--@ 14 user staff  2 Feb 16:03 -- .tool-versions
❯ cat .envrc
# modifying the path relevant for the test case
PATH_add node_modules/.bin
export FIRST='first'
❯ cat .tool-versions
nodejs 17.7.1

# as expected, rtx added local nodejs version 17.7.1 to PATH
❯ echo $PATH
/Users/user/.local/share/rtx/installs/nodejs/17.7.1/bin:...
❯ cd ..
rtx: nodejs@system
direnv: unloading
❯ ll
drwxr-xr-x@    - user staff  2 Feb 09:54 -- load-first

# local rtx version from previous directory is still in PATH, not system version from nearest ancestor
❯ echo $PATH
/Users/user/.local/share/rtx/installs/nodejs/17.7.1/bin:...
❯ node -v
v17.7.1

Jeff Dickey · Answer 34 · Thu Feb 02 2023 23:51:08 GMT+0800 (China Standard Time)

thanks for the E2E test, I think it's fine for now to keep everything consolidated in this issue. I think the nodejs system bit here is actually a no-op. It would probably behave like this even if there wasn't a tool-versions file in the parent directory (as long as no higher-up directory has a nodejs version set).

But yeah, rtx adds things to PATH and DIRENV_DIFF, when you navigate away, it removes it from PATH but not from DIRENV_DIFF so direnv is adding back things that should have been removed. This is a relatively easy fix, I just need to remove the installs from DIRENV_DIFF too and I already have a library that does that, I just need to call it.

I think, based on my knowledge of the internals, there is another potential bug people could be surprised by. If you manually set export PATH="$HOME/.local/share/rtx/installs/nodejs/..., then rtx will actually remove that because it removes everything with that prefix every time it runs and only adds the active versions. I think we will actually want to be smarter than that and remove only PATH elements that were previously set. This isn't trivial, I'll have to add a new field in RTX_DIFF that tracks the PATH modifications and go through them one-by-one, for both PATH and DIRENV_DIFF.

I think it makes sense to make both changes at the same time since otherwise I'd have to put some naive logic in for just DIRENV_DIFF and replace it later.

Thanks again for your help with this @delicb @amoosbr. I think we're getting close here.

Jeff Dickey · Answer 35 · Fri Feb 03 2023 09:27:22 GMT+0800 (China Standard Time)

here's the next steps I think:

add individual path entires that were added to __RTX_DIFF
stop removing all paths with ~/.local/share/rtx/installs when loading "pristine" path and instead remove only those listed in __RTX_DIFF as "new" (new in this case means in the environment when it loads, so "old")
remove paths listed in __RTX_DIFF from DIRENV_DIFF

in more plain english, what this change will do is track the entries in PATH that were added when rtx was loaded. Then the next time that rtx loads it first removes these entries from PATH and DIRENV_DIFF. That was it is starting from a state as if rtx hadn't run to begin with. It's basically a rollback to get the environment variables back into the state they were before it was run.

Hopefully I can get around to this soon.

Jeff Dickey · Answer 36 · Mon Feb 06 2023 10:42:12 GMT+0800 (China Standard Time)

1.5.0 is out which makes the incremental changes to PATH and DIRENV_DIFF. That e2e test is passing now.

I'd like to close this but I also anticipate that one of you will uncover more behavior that still isn't quite right so I'll wait.

Alexander Moosbrugger · Answer 37 · Mon Feb 06 2023 17:29:30 GMT+0800 (China Standard Time)

In my initial test this morning the issues, I wasn't able to reproduce any of the issues mentioned before.

I just add here a few comments for usage reference:

Hook order

In my experience, its still relevant and for zsh should look like:
.zshrc:

eval "$(direnv hook zsh)"
eval "$(rtx activate zsh)"

For zsh, you can verify hook order via echo $chpwd_functions and _rtx_hook should be listed before _direnv_hook.

❯ echo $chpwd_functions
_rtx_hook _direnv_hook

Perhaps in a documentation update, when this issue gets close?

PATH order gets modified and can break custom dev workflows

I have no idea, how common these scenarios are and if you @jdxcode think, they are working as expected, are to worked on within this issue or should be in their own new issue to collect feedback. I can also try to create an e2e test case, if the order should be kept.
I think keeping the PATH order (what was added before rtx paths and after rtx) would avoid scenarios 1. and 2. If it's how rtx should behave. Example 3. is something, I think rtx can not do much about.

.envrc used in samples:

layout node

Install of new tool, changes PATH order

A bit interesting for a user to predict. It's probably related to rtx not keeping the PATH order (before/after rtx) mentioned in this comment and not related just to direnv.

-> workaround change out of the direnv managed directory and back in

# add local node_modules bin to front of PATH, to use project binary versions instead of global ones
❯ echo $PATH
/Users/user/Projects/tmp/rtx-direnv-path-order/node_modules/.bin:/Users/user/.local/share/rtx/installs/nodejs/18.0.0/bin:...

# install some new tool versions (not used in the local project)
❯ rtx i nodejs@18.13.0
✓ Runtime nodejs@18.13.0 installed
rtx: nodejs@18.0.0

# path order changed. Now using global node package binaries first
❯ echo $PATH
/Users/user/.local/share/rtx/installs/nodejs/18.0.0/bin:/Users/user/Projects/tmp/rtx-direnv-path-order/node_modules/.bin:...

Similar behavior, when changing to a new directory with new .tool-versions file and back to the previous directory

When traversing to subproject with new .tool-versions file. Which tool added, is irrelevant.
-> workaround, where needed: add .envrc with i.e. source_up_if_exists or change out of the direnv managed directory and back in

# add local node_modules bin to front of PATH, to use project binary versions instead of global ones
❯ echo $PATH
/Users/user/Projects/tmp/rtx-direnv-path-order/node_modules/.bin:/Users/user/.local/share/rtx/installs/nodejs/18.0.0/bin:...

# path order changed. Now using global node package binaries first
❯ cd test-with-new-tool-versions
rtx: python@3.8.10 nodejs@18.0.0
❯ echo $PATH
/Users/user/.local/share/rtx/installs/python/3.8.10/bin:/Users/user/.local/share/rtx/installs/nodejs/18.0.0/bin:/Users/user/Projects/tmp/rtx-direnv-path-order/node_modules/.bin:...

# switching back to initial directory keeps the PATH order. Still using global node package binaries first
❯ cd ..
rtx: python@3.10.9 nodejs@18.0.0
❯ echo $PATH
/Users/user/.local/share/rtx/installs/python/3.10.9/bin:/Users/user/.local/share/rtx/installs/nodejs/18.0.0/bin:/Users/user/Projects/tmp/rtx-direnv-path-order/node_modules/.bin:...

On update local tool version

This will anyway be complicated to automate in combination with direnv, as direnv doesn't watch .tool-versions file.
-> If common, add watch_file to local .envrc file or useuse rtx) -> Corner case, that a user can decide, how to handle

❯ echo $PATH
/Users/user/Projects/tmp/rtx-direnv-path-order/node_modules/.bin:/Users/user/.local/share/rtx/installs/nodejs/18.0.0/bin:...
❯ rtx local nodejs@18.13.0
nodejs 18.13.0
rtx: nodejs@18.13.0
❯ echo $PATH
/Users/user/.local/share/rtx/installs/nodejs/18.13.0/bin:/Users/user/Projects/tmp/rtx-direnv-path-order/node_modules/.bin:...

Jeff Dickey · Answer 38 · Thu Feb 09 2023 02:49:56 GMT+0800 (China Standard Time)

added more information around this to the readme in bb61bc5. I'm probably going to focus on other things (like rtx where) when I can focus on rtx development for now. If people are having issues like with what @amoosbr has outlined above (or other issues) go ahead and comment on this ticket.

I think we've sorted out the major issues with rtx activate and direnv hook for now.

Jeff Dickey · Answer 39 · Thu Feb 09 2023 22:28:44 GMT+0800 (China Standard Time)

I was thinking about all of this this morning and I feel like the fact that you're doing layout python and also using rtx to manage the python runtime feels a bit broken. I'm not sure if "broken" here means that rtx should be doing more, or you should be using rtx in a different way, but something feels very off to me. That said, I really don't feel I have a very good understanding about virtualenvs especially how they integrate (or don't integrate) with things like pyenv, pipenv, poetry, etc.

Because rtx is capable of modifying environment variables for all commands, not just in shims, I wonder if there could be an rtx python plugin that could do more here so that it wouldn't be necessary to also use direnv for configuring the virtualenv.

If someone knows a lot more about Python than I do, perhaps you could offer a suggestion on what might be possible here.

One of my first designs of rtx was that it was actually a mix of direnv and asdf and it used a different config file than .tool-versions that could set arbitrary environment variables in directories. I later ditched that just to simplify things (.tool-versions was always supported), but we could bring this config back in for sure. I think a lot of the code for it still exists actually.

Jan Katins · Answer 40 · Fri Feb 10 2023 05:10:04 GMT+0800 (China Standard Time)

I was thinking about all of this this morning and I feel like the fact that you're doing layout python and also using rtx to manage the python runtime feels a bit broken

I kind of agree with this: I would always recommend creating a venv for a project and that means the rtx supplied python will actually be overwritten by the one from the venv. The only time you need one is when you have to create the venv. For that I now create my venvs like that: rtx x python@$(rtx latest python)) -- python3 -m venv ..

To make this nicer in direnv, I will probably build something like use rtx python@3.11 python3 which would add that to the path via basename $(rtx x ${1} -- which ${2}) (or do some magic with export RTX_PYTHON_VERSION=3.xx and rtx env). And then use my regular use poetry to get the venv created and activated.

UPDATED:
What I would love for this use case (only use .envrc instead of .tool-versions) is a command which I could eval and which would only activate a specific app: e.g. eval "$(rtx env --manual python@3.11)"

Jeff Dickey · Answer 41 · Fri Feb 10 2023 05:18:52 GMT+0800 (China Standard Time)

what about export PATH="$(rtx where python@3.11)/bin:$PATH", does that fit your use-case?

for some runtimes this might be more challenging since they have more than just /bin, but with python I feel this should be simple enough

Jan Katins · Answer 42 · Fri Feb 10 2023 05:22:26 GMT+0800 (China Standard Time)

what about export PATH="$(rtx where python@3.11)/bin:$PATH", does that fit your use-case?

Yes, python should work that way. For others, the eval $(rtx env --manual python@3.11) idea would work better?

Jeff Dickey · Answer 43 · Fri Feb 10 2023 05:23:38 GMT+0800 (China Standard Time)

I'd like to see some use cases about specific languages. I feel this isn't likely to even come up since nothing else really has virtualenv-like behavior. (Though I'm happy to be corrected on this point, I don't know every language)

Jan Katins · Answer 44 · Fri Feb 10 2023 05:29:27 GMT+0800 (China Standard Time)

Rust sets two more env variables:

λ  rtx env
export CARGO_HOME=/Users/jankatins/.local/share/rtx/installs/rust/1.67.0
export RUSTUP_HOME=/Users/jankatins/.local/share/rtx/installs/rust/1.67.0
export PATH=...

Basically every asdf plugin with an exec-env would need more than PATH.

-> It's not a virtualenv scenario, but use rtx rust@1.63 would do the right thing. Basically the direnv section in the readme would go to "If you use direnv and need to activate specific versions, use use rtx plugin@version in .envrc and do NOT create a local .tool-versions file".

Jeff Dickey · Answer 45 · Fri Feb 10 2023 05:34:33 GMT+0800 (China Standard Time)

What I'm not convinced of is that people would need to run rtx env for a single runtime for anything other than Python. Python and its virtualenvs is a unique use-case where rtx env --manual would be helpful but I'm not (yet) convinced there are more.

I get that it would be somewhat of a pain to setup just rust by itself but I'm not sure anyone would have a reason for wanting to do that in the first place. If you just want to configure a single runtime, then why are there other runtimes in .tool-versions anyways?

BTW I'm not happy with rtx/asdf's integration with rust anyways but that's a separate discussion. I think rtx should integrate with rustup directly rather than managing rust compilers itself.

Alexander Moosbrugger · Answer 46 · Fri Feb 10 2023 15:44:52 GMT+0800 (China Standard Time)

I'm not sure, I could follow the discussion. But related to

I feel like the fact that you're doing layout python and also using rtx to manage the python runtime feels a bit broken

I don't see rtx and direnv both managing the actual python version. With the right hook order, the python version is managed by rtx - .tool-versions and direnv - layout python3 manages just the virtualenv setup.
This combination does, in my understanding, what Jan is doing manually. Direnv, in this case, just does an opinionated virtualenv setup for you. But I can be wrong.

PreRequirement:

When switching to a project directory the rtx hook is executed first and afterwards the direnv hook runs
- With asdf this was of course not relevant (shims). It is also not relevant, when we use use rtx and have rtx hook deactivated

Correct Sequence with the hooks running in the correct order:

change to project directory
rtx hook runs and modifies the PATH to point to python version configured in local .tool-versions
- .tool-versions is managed by git in our projects to define the python3 version for the project
- PATH after this step would look like: /Users/user/.local/share/rtx/installs/python/3.8.10/bin:...
Direnv hook runs local .envrc uses PATH (python3) pointing to python version set above and creates virtualenv
- .envrc is not managed by git in our projects, as every dev can do this as he likes. But the below setup is common.
- layout python3 uses the python3 binary found in the PATH
  - With the correct rtx/direnv hook order, it is pointing to the locally configured python binary from .tool-versions
- layout python3 will create (or reuse) a virtualenv for that python version and set VIRTUAL_ENV
  - VIRTUAL_ENV=/Users/user/Projects/tmp/rtx-project/.direnv/python-3.8.10
- layout python3 will add the python bin folder from the virtualenv to the front of PATH
  - PATH=/Users/user/Projects/tmp/rtx-project/.direnv/python-3.8.10/bin:...
After the hooks ran, the environment looks like
- echo $VIRTUAL_ENV=/Users/user/Projects/tmp/rtx-project/.direnv/python-3.8.10
- echo $PATH=/Users/user/Projects/tmp/rtx-project/.direnv/python-3.8.10/bin:/Users/user/.local/share/rtx/installs/python/3.8.10/bin:...
In this case, I see a clear separation of responsibility between rtx and direnv
- rtx sets the python3 version to use
- direnv with layout python3 creates an opinionated virtualenv environment using the rtx version

In this scenario, if the direnv hook would run first it would find the "wrong" python binary in the PATH

Sequence with hooks running in the "wrong" order:

global python version set to 3.10.10
- PATH after this step would look like: /Users/user/.local/share/rtx/installs/python/3.10.10/bin:...
change to project directory
direnv hook runs and find python3 binary on PATH pointing to 3.10.10
- layout python3 creates a virtualenv using global python3 binary on PATH
- VIRTUAL_ENV=/Users/user/Projects/tmp/rtx-project/.direnv/python-3.10.10
- PATH=/Users/user/Projects/tmp/rtx-project/.direnv/python-3.10.10/bin:...
rtx hook runs and configures the PATH based on local .tool-versions
- After hook: PATH=/Users/user/.local/share/rtx/installs/python/3.8.10/bin:/Users/user/Projects/tmp/rtx-project/.direnv/python-3.10.10/bin:...
After booth hooks ran, the local setup is misconfigured
- echo $VIRTUAL_ENV=/Users/user/Projects/tmp/rtx-project/.direnv/python-3.10.10
- echo $PATH=/Users/user/.local/share/rtx/installs/python/3.8.10/bin:/Users/user/Projects/tmp/rtx-project/.direnv/python-3.10.10/bin:...

Not sure, if the discussion was about the hook order requirement or my usage of direnv, rtx, and python, but I hope I could help clarify some of your thoughts.

But the hook order is not only related just to python. Also valid for every other direnv use case modifying the PATH, where the PATH order is relevant. That's why I used a nodejs sample in this comment.

Jeff Dickey · Answer 47 · Fri Feb 10 2023 22:58:18 GMT+0800 (China Standard Time)

What I don't like is that the virtualenv contains a python binary that gets added to PATH ahead of rtx. That means the order matters, and it makes the potential for conflicts possible.

I'm not sure how feasible this would be, but I wonder if rtx could set the virtualenv up so direnv wouldn't have to. That way we wouldn't need to prescribe a specific order.

But the hook order is not only related just to python. Also valid for every other direnv use case modifying the PATH, where the PATH order is relevant. That's why I used a nodejs sample in this #8 (comment).

The thing is that should be a pretty esoteric use-case: managing the path to node with direnv and with rtx. The thing about virtualenvs is it isn't esoteric at all, it's probably pretty common. I'm relatively ok with a rare use-case having requirements like the ordering of activation mattering, but not for a common one.

Alexander Moosbrugger · Answer 48 · Fri Feb 10 2023 23:20:07 GMT+0800 (China Standard Time)

Thanks for clarifying your point. Now I understood your thoughts better.

But I don't have a good solution. Personally, I'm not convinced yet, that rtx should start handling additional use cases like the python virtualenv setup. I like that rtx (or asdf) manage 'just' the correct tool version and other tools like direnv, poetry, ... can use the provided tool versions and can focus on the setup of the development environment.

But I can from a usability perspective understand, that the requirement on a correct order of hooks, is not perfect and might lead to complains with users not reading the documentation in detail. But for these use cases, I think rtx with shims, would be the better solution. Because even if rtx now sets up virtualenvs, you still might run into strange behavior, as rtx boosts it's own PATHs to the front.

But perhaps you or someone else has a great idea to solve this.

Jeff Dickey · Answer 49 · Sat Feb 11 2023 00:20:43 GMT+0800 (China Standard Time)

agreed on all points. I think a shim mode may make sense, it wouldn't be that hard to do. I wouldn't make it the default of course.

Jeff Dickey · Answer 50 · Mon Feb 27 2023 06:07:56 GMT+0800 (China Standard Time)

I think we can safely close this now as I'm not hearing reports about any more issues with direnv. Happy to reopen if people have any.

I've also added experimental shims support in #213 which provides an alternative way to use rtx.

Jeff Dickey · Answer 51 · Tue Feb 28 2023 22:19:22 GMT+0800 (China Standard Time)

for people coming across this: you can use direnv with rtx in 2 ways: with rtx activate and rtx direnv activate.

The first is what I would recommend. If you need the rtx runtimes loaded in direnv (such as when using layout python), then just call eval "$(rtx env -s bash)" in .envrc. It shouldn't matter in that case if direnv is activated before or after rtx.

There is also rtx direnv activate (see help for this command on how to use) which allows you to use rtx from inside direnv. This has less functionality but is possibly more compatible. In this mode, rtx is called by direnv inside of the envrc file so rtx does not need to do any stateful PATH and env var management.

The downside of rtx direnv activate is that you will likely need to have an envrc file next to any .tool-versions files or else direnv won't know to call rtx.

Evgeny · Answer 52 · Tue Feb 28 2023 22:40:18 GMT+0800 (China Standard Time)

think there is a typo: you're recommending the second rtx activate, not the first rtx direnv activate

Jeff Dickey · Answer 53 · Wed Mar 01 2023 00:09:53 GMT+0800 (China Standard Time)

I did notice a bit of strange behavior just now:

~ ❯ which python
/Users/jdx/.direnv/python-3.11.2/bin/python
~ ❯ touch .tool-versions
~ ❯ which python
/Users/jdx/.local/share/rtx/installs/python/3.10.10/bin/python

if you run touch .tool-versions that triggers rtx to run again after direnv, which isn't the way we generally want it to work. However this is happening because rtx needs to reload but direnv does not. If I ran touch .tool-versions .envrc it would behave as expected.

However at least in the case of python I don't know if this would have practical implications, the same python version is used, it's just referenced from a different PATH. I figured I would mention it though.

If your setup is sensitive to this sort of thing, use rtx direnv activate which won't have any ordering issues like this.

Jeff Dickey · Answer 54 · Wed Mar 01 2023 00:44:14 GMT+0800 (China Standard Time)

changed my recommendation to add eval "$(rtx env)" in .envrc. It didn't seem like for me having direnv loaded before or after actually works correctly anymore. Adding the eval seems to do the right thing though.

Jeff Dickey · Answer 55 · Thu Mar 02 2023 08:21:34 GMT+0800 (China Standard Time)

for layout python users: I just added support for virtualenv's from within rtx and the rtx-python plugin. See docs for more: https://github.com/jdxcode/rtx-python#virtualenv-support

I think this will work better than using layout python to try to manage the virtualenv (which has its own runtime) while also using rtx to manage the runtime only.

I'm no python expert though, so let me know if it works ok. I looked at the direnv source and layout python isn't very complicated though.

Bojan Delić · Answer 56 · Sun Mar 05 2023 10:11:34 GMT+0800 (China Standard Time)

@jdxcode I quickly checked rtx-python plugin fork. I found two problems:

It does create virtualenv, but it does not deactivate it when directory is changed. Actually, it is very sticky and navigating to other directories and even reloading shell kept created virtualenv first in path. Eventually, I managed to deactivate env with:

$ rtx deactivate
$ set -e __RTX_DIFF
$ set -e __RTX_WATCH
$ rtx cache clean
$ rtx activate fish | source

It might be that only subset of these is needed, but I tried couple of combinations and virtualenv kept being activated until I executed all of these, but since I am not sure why this happens, I am not sure if all of these are required.

Virtualenv is created in subdirectory if it is cd-ed directly. E.g. if I have ~/myproj that contains file .rtx.toml and directory subdir and I do cd myproj/subdir, it creates virtualenv in subdir, when it should be in project root.

I can open ticket in https://github.com/jdxcode/rtx-python if you wish to track it there, but I think replacing layout python will be more involved than current implementation of exec-env. I would love to use virtualenv parameter in .rtx.toml, but for example I have override for direnv_layout_dir to create virtualenvs in $XDG_CACHE_HOME/direnv/layouts/<project-name>, which direnv supports out of the box. There are probably more such features which would complicate python asdf plugin.

Bojan Delić · Answer 57 · Sun Mar 05 2023 10:15:23 GMT+0800 (China Standard Time)

However at least in the case of python I don't know if this would have practical implications, the same python version is used, it's just referenced from a different PATH. I figured I would mention it though.

Python version is the same, but environment might not be. E.g. I keep global python environment (in ~/.local/share/rtx/installs/python) clean and without any libraries (well, I might have some, but not project specific), while for each project I have virtualenv referencing global environment and each has its own set of packages installed. So, same version does not mean much when all the dependencies are different.

I understand what you meant - for Ruby, Go - it does not matter. But for Python it does, since all dependencies are within virtualenv.

Jeff Dickey · Answer 58 · Mon Mar 06 2023 00:20:09 GMT+0800 (China Standard Time)

I've got a fix for most of these, but the "stickiness" I haven't quite figured out yet. I only looked at it for a few minutes last night though. I think I have a idea if generally why it might happen though. I'll see if I can fix it later today.

This might involve some back and forth just because I'm not a great tester but I am pretty sure we can get there.

Jeff Dickey · Answer 59 · Mon Mar 06 2023 06:05:30 GMT+0800 (China Standard Time)

Python version is the same, but environment might not be. E.g. I keep global python environment (in ~/.local/share/rtx/installs/python) clean and without any libraries (well, I might have some, but not project specific), while for each project I have virtualenv referencing global environment and each has its own set of packages installed. So, same version does not mean much when all the dependencies are different.

I mean, now that rtx has support for venv's directly this might be a moot point, but I still think this shouldn't be a problem. With what I proposed: which python might be indeterministic—however VIRTUAL_ENV would not be. So unless python ignores VIRTUAL_ENV and looks at the directory it is called in with $0, it shouldn't be a problem.

In other words, you might encounter the following:

$ which python
~/.rtx/installs/python/3.11/bin/python
$ echo $VIRTUAL_ENV
~/src/myproj/.venv

I haven't tested this, but wouldn't this still cause pip to install dependencies to ~/src/myproj/.venv and keep ~/.rtx/installs/python/3.11 clean?