racket-repl exec-path does not match racket-mode buffer's exec-path

Question

racket-repl exec-path does not match racket-mode buffer's exec-path

KarlJoad opened this issue 2 months ago · comments

I do not have racket installed globally, but instead have it edited into my path by direnv when I cd into the project.

I couple direnv with envrc in Emacs so that buffers have their process-environment and exec-path buffer-local variables changed for buffers that are currently inside of the project.

Everything has been working fine and correctly so far, and this works for other modes (like julia-mode coupled with julia-repl), so it is not an issue with my Emacs configuration (as far as I can tell). But when I do this same step in racket-mode, I get the following error (and a Racket REPL buffer that is not connected to a REPL).

racket--back-end-args->command: Cannot find Racket executable
racket-program: "racket"
exec-path: ("/home/karljoad/.nix-profile/bin" "/home/karljoad/.guix-home/profile/bin" "/home/karljoad/.guix-home/profile/sbin" "/run/setuid-programs" "/home/karljoad/.config/guix/current/bin" "/home/karljoad/.guix-profile/bin" "/run/current-system/profile/bin" "/run/current-system/profile/sbin" "/gnu/store/prp5j7kxib08xdimdzjb3m3jq4sm9s0i-gzip-1.12/bin" "/gnu/store/dy798in3s7nwcjcyvv93hxlbfd01f0vn-coreutils-9.1/bin" "/gnu/store/fl7s045xbwzk8prx9g0jkais2sd209qz-emacs-pgtk-29.1/libexec/emacs/29.1/x86_64-pc-linux-gnu")

However, the racket-mode buffer has the correct exec-path

;; Inside of a racket-mode buffer, which has had exec-path modified by envrc
(message "%s" exec-path)
(/gnu/store/7k59cpzy5jh7ma7432spdx2c6nkr7jc4-profile/bin/ /home/karljoad/.nix-profile/bin/ /home/karljoad/.guix-home/profile/bin/ /home/karljoad/.guix-home/profile/sbin/ /run/setuid-programs/ /home/karljoad/.config/guix/current/bin/ /home/karljoad/.guix-profile/bin/ /run/current-system/profile/bin/ /run/current-system/profile/sbin/ /gnu/store/prp5j7kxib08xdimdzjb3m3jq4sm9s0i-gzip-1.12/bin/ /gnu/store/dy798in3s7nwcjcyvv93hxlbfd01f0vn-coreutils-9.1/bin/)

;; Still inside of that racket-mode buffer
(shell-command "racket --version")
;; Or invoke shell-command with M-! (default keybinding)
;; The version of racket is printed, and it is the correct version
;; Welcome to Racket v8.11.1 [cs].

The first entry in exec-path is where racket is installed:

$ ls /gnu/store/7k59cpzy5jh7ma7432spdx2c6nkr7jc4-profile/bin/
... racket ...

Digging in, the problem appears to finally appear from racket--cmd-open, because if I do (message "exec-path as seen from racket--cmd-open: %s" exec-path) and redefine racket--cmd-open to now include that message call, I get:

exec-path as seen from racket--cmd-open: (/home/karljoad/.nix-profile/bin /home/karljoad/.guix-home/profile/bin /home/karljoad/.guix-home/profile/sbin /run/setuid-programs /home/karljoad/.config/guix/current/bin /home/karljoad/.guix-profile/bin /run/current-system/profile/bin /run/current-system/profile/sbin /gnu/store/prp5j7kxib08xdimdzjb3m3jq4sm9s0i-gzip-1.12/bin /gnu/store/dy798in3s7nwcjcyvv93hxlbfd01f0vn-coreutils-9.1/bin /gnu/store/fl7s045xbwzk8prx9g0jkais2sd209qz-emacs-pgtk-29.1/libexec/emacs/29.1/x86_64-pc-linux-gnu)
racket--back-end-args->command: Cannot find Racket executable
racket-program: "racket"
exec-path: ("/home/karljoad/.nix-profile/bin" "/home/karljoad/.guix-home/profile/bin" "/home/karljoad/.guix-home/profile/sbin" "/run/setuid-programs" "/home/karljoad/.config/guix/current/bin" "/home/karljoad/.guix-profile/bin" "/run/current-system/profile/bin" "/run/current-system/profile/sbin" "/gnu/store/prp5j7kxib08xdimdzjb3m3jq4sm9s0i-gzip-1.12/bin" "/gnu/store/dy798in3s7nwcjcyvv93hxlbfd01f0vn-coreutils-9.1/bin" "/gnu/store/fl7s045xbwzk8prx9g0jkais2sd209qz-emacs-pgtk-29.1/libexec/emacs/29.1/x86_64-pc-linux-gnu")

I saw the discussion on purcell/envrc#22 and #539, which resolves this issue.

I was wondering what would be required to have to skip the (racket-start-back-end) step when switching projects?

Greg Hendershott · Answer 1 · Thu Apr 04 2024 19:56:24 GMT+0800 (China Standard Time)

Caveat: I'm still inhaling coffee, as well as reloading my brain with some of these details from over the years.

Something new since #539 is I added support for multiple simultaneous back ends, as discussed here.

The main motivation was to support remote back ends via tramp. (Where the .rkt path is something like /user@host:xxx.)
Another motivation was to use different versions of Racket for .rkt files under certain subdirs of /, whether local or remote. (The default back end is everything under local /.)

The latter is closer to what you want, IIUC?

The catch is you need to use racket-add-back-end in your Emacs init file to say what subdir trees use which back ends -- not by using .envrc files, direnv, and envrc. So this might:

Feel bad because it's doing a similar kind of configuration in two different places/ways?
Just not work at all because you have something happening for nix where the .envrc files get updated, idk?

Anyway, once I understand more clearly, maybe there could be some way to have back ends automagically configured by .envrc files?? Which I think might be exactly-ish what you want?

Karl H · Answer 2 · Fri Apr 05 2024 00:44:21 GMT+0800 (China Standard Time)

The latter option (use different Racket versions locally at different subdirs) is exactly what I want.

racket-add-back-end works, but putting that in my init.el feels a little incorrect, like you said. It might be possible to hack around putting this in my init.el by putting it in a .dir-locals.el and use the eval escape hatch.

((nil . ((eval . (racket-add-back-end (file-name-directory (pwd)))))))
;; This errors with the message "racket-add-back-end: racket-add-back-end: directory must be file-name-absolute-p"

I will look over the documentation you posted to understand why racket-mode uses back-ends. This might just me being too used to other Lisps where replacing the REPL is a simple matter of changing which binary is run.

It is not that I want direnv to change/configure the back-end used. I just use direnv to make racket, as a program, available to me for use (I do not want racket installed globally).

I think a combination of .envrc & direnv making programs available and .dir-locals.el configuring them may be the right way to do this for now.

Greg Hendershott · Answer 3 · Fri Apr 05 2024 01:40:34 GMT+0800 (China Standard Time)

((nil . ((eval . (racket-add-back-end (file-name-directory (pwd)))))))
;; This errors with the message "racket-add-back-end: racket-add-back-end: directory must be file-name-absolute-p"

This error is probably because pwd prints default-directory but returns nil?

Maybe you'd want just (racket-add-back-end default-directory) in the dir-locals.el?

As for why can't we just use the direnv approach, I think the answer is "lifetimes".

The direnv way is great for setting up env vars like PATH for use by various ad hoc shell commands.

However the back end for Racket Mode is used to support a wide variety of commands that can't be implemented in Emacs, and need a running Racket process. There are good reasons for this to be a long-running Racket process:

Racket could take a few seconds to start itself, and load the back end code.
Some commands benefit from caching state (about completions, docs, etc.).
A few commands can be very frequent (at one extreme, racket-hash-lang-mode will issue a command for every edit change you make, e.g. every keystroke).

And so the design assumes this long-running back end.

Even things like multiple REPLs are run out of this one process.

Even when there are multiple back end processes (local vs. remote, and/or for different directory sub trees), the assumption is still that they are started when first needed by some command, and remain running.

So this isn't quite the run-frequent-shell-commands idea about lifetimes. Maybe I'm being dense, but I don't immediately see how to neatly reconcile these ideas. I'm sure it's possible, somehow, but I'm not sure at what cost of complexity and reliability.

So TL;DR if you could make the racket-add-back-end approach work, with say dir-locals, that is the simplest/quickest way forward I see right now. (But as always, happy to think about it more, especially if that turns out to be annoying for you.)

Greg Hendershott · Answer 4 · Tue Apr 09 2024 02:17:53 GMT+0800 (China Standard Time)

Although I think the above is a not-terrible resolution, this has been nagging me for a few days: Is there some day I could avoid you needing to use dir-locals.el at all?

One idea: A function for envrc-mode-hook, something like this:

(defun racket-back-end-envrc-mode-hook ()
  "A value for `envrc-mode-hook'.

When envrc is being used, check whether `exec-path' and
`process-environment' match for the current buffer and the
current back end process buffer. If they do not, add a new back
end for the directory of the dominating envrc file, if any. As a
result, the new back end will use values from `exec-path' and
`process-environment'."
  (when (featurep 'envrc)
    (declare-function envrc--find-env-dir "envrc" ())
    (require 'envrc)
    (let ((back-end (racket-back-end)))
      (unless (equal (cons exec-path process-environment)
                     (with-current-buffer (racket--back-end-process-name back-end)
                       (cons exec-path process-environment)))
        (when-let ((dir (envrc--find-env-dir)))
          ;; Unless we already have a back end for `dir', add one.
          (unless (cl-find dir
                           racket-back-end-configurations
                           :test
                           (lambda (dir back-end)
                             (equal dir (plist-get back-end :directory))))
            (racket-add-back-end dir)))))))

And so the only config you would need to do is, once/generally in your Emacs init file, (add-hook 'envrc-mode-hook #'racket-back-end-envrc-mode-hook).

Thereafter, whenever the envrc-mode minor mode is activated in each new buffer, this check would happen automatically. (That is frequent enough to be effective, without being excessively frequent, IIUC.)

Although the above byte compiles cleanly, I haven't tested yet. I'd need to add envrc package and set up some test directories. I plan to do that, soon-ish. But if you were in the mood to go ahead and try this yourself, earlier, I wanted to point this out.

Greg Hendershott · Answer 5 · Tue Apr 09 2024 22:29:30 GMT+0800 (China Standard Time)

No that's not quite right yet.

I have all the things (direnv, envrc-mode) installed now; will work this hands-on and not bother you again until/unless I have a solution...

Karl H · Answer 6 · Tue Apr 09 2024 23:20:06 GMT+0800 (China Standard Time)

No worries. I am using both Guix and Nix, so if you want me to test anything for those package managers as well, let me know!

Thanks for all of your help!

Greg Hendershott · Answer 7 · Wed Apr 10 2024 20:00:13 GMT+0800 (China Standard Time)

I have a commit that works for me in a test scenario with direnv, envrc-mode, and a couple test dirs with .envrc files.

The basic idea is to add a function to envrc-mode-hook. So it runs whenever envrc-mode minor mode initializes in a buffer, and after it's read the .envrc if any, set the buffer-local values for exec-path and process-shell, and remembered the .envrc dir.

We add a distinct Racket Mode back end, if we don't yet have one, for the dir. (You don't need to manually put a parallel dir-locals.el in every .envrc directory, just to add a back end. You don't even need to do the add-hook, it arranges that.)

So this happens per buffer init, which is a reasonable frequency (unlike per Racket Mode command, which would have been much too frequent).

Above are some commits with CI failures where I was trying to supplicate the byte-compiler and check-declare for the scenarios where envrc-mode is installed or not, both. Any of them should have worked for you (this was a "warnings as errors" kind of "failure"), but the last one, commit 2d5207e is what I plan to merge.

Would you like to try it and confirm, before I merge?

(If not convenient for you to do so, within a few days, I can merge anyway.)

Karl H · Answer 8 · Thu Apr 11 2024 00:06:27 GMT+0800 (China Standard Time)

Awesome! Thanks for taking care of this so quickly! I just did a quick test (installed through elpaca to the 2d5207e commit) and everything worked nicely.

I am getting some errors: File mode specification error, an error with racket--logger-on-notify. I am including the excerpts from my *Messages* buffer. (Ignore the quail stuff. It is to get Greek letters from typing their English names.)

Getting direnv to run in a project and have envrc run on an already-existing file.

Running direnv in /home/karljoad/Repos/cs_424-dynamics_pl/ ... (C-g to abort)
Direnv succeeded in /home/karljoad/Repos/cs_424-dynamics_pl/
Running direnv in /home/karljoad/Repos/cs_424-dynamics_pl/ ... (C-g to abort)
Direnv succeeded in /home/karljoad/Repos/cs_424-dynamics_pl/
Loading quail/latin-ltx (native compiled elisp)...done
File mode specification error: (user-error Cannot find Racket executable
racket-program: "racket"
exec-path: ("/home/karljoad/.nix-profile/bin" "/home/karljoad/.guix-home/profile/bin" "/home/karljoad/.guix-home/profile/sbin" "/run/setuid-programs" "/home/karljoad/.config/guix/current/bin" "/home/karljoad/.guix-profile/bin" "/run/current-system/profile/bin" "/run/current-system/profile/sbin" "/gnu/store/prp5j7kxib08xdimdzjb3m3jq4sm9s0i-gzip-1.12/bin" "/gnu/store/dy798in3s7nwcjcyvv93hxlbfd01f0vn-coreutils-9.1/bin" "/gnu/store/fl7s045xbwzk8prx9g0jkais2sd209qz-emacs-pgtk-29.1/libexec/emacs/29.1/x86_64-pc-linux-gnu"))
racket-back-end-envrc-mode-hook configuring back end for "/home/karljoad/Repos/cs_424-dynamics_pl/" to use "/gnu/store/sbi7k0ahq3dfdhk5l1drrrv0jknrhl2s-profile/bin/racket"

Direnv running in a new project, opening a new file (did not previously exist).

Running direnv in /home/karljoad/Repos/disentanglement-redex/ ... (C-g to abort)
Direnv succeeded in /home/karljoad/Repos/disentanglement-redex/
racket-back-end-envrc-mode-hook configuring back end for "/home/karljoad/Repos/disentanglement-redex/" to use "/gnu/store/sbi7k0ahq3dfdhk5l1drrrv0jknrhl2s-profile/bin/racket"
Error running timer ‘racket--logger-on-notify’: (user-error "Cannot find Racket executable
racket-program: \"racket\"
exec-path: (\"/home/karljoad/.nix-profile/bin\" \"/home/karljoad/.guix-home/profile/bin\" \"/home/karljoad/.guix-home/profile/sbin\" \"/run/setuid-programs\" \"/home/karljoad/.config/guix/current/bin\" \"/home/karljoad/.guix-profile/bin\" \"/run/current-system/profile/bin\" \"/run/current-system/profile/sbin\" \"/gnu/store/prp5j7kxib08xdimdzjb3m3jq4sm9s0i-gzip-1.12/bin\" \"/gnu/store/dy798in3s7nwcjcyvv93hxlbfd01f0vn-coreutils-9.1/bin\" \"/gnu/store/fl7s045xbwzk8prx9g0jkais2sd209qz-emacs-pgtk-29.1/libexec/emacs/29.1/x86_64-pc-linux-gnu\")")

Greg Hendershott · Answer 9 · Thu Apr 11 2024 03:18:48 GMT+0800 (China Standard Time)

Hmm.

Re the file-mode-specification error for an existing file: I can't repro this (assuming I understand the scenario correctly). From your *Messages* it seems like something is running a back end command before the envrc-mode-hook has run and configured the dir specific back end. But I'm not sure what something that is (do you have racket-xp-mode enabled? OK but so do I), or why the order is that way. Maybe I can repro this if I set racket-program to something that doesn't exist (to be like you, where there is no racket on the "global" PATH, IIUC).
Re the racket--logger-on-notify error for a new file: Also can't repro, yet. Here the envrc specific back end message is before the error, unlike 1, and that's counter-intuitive. But maybe here too I need a non-existing global racket-program to shed more light.

So I'll investigate...

Greg Hendershott · Answer 10 · Thu Apr 11 2024 19:35:00 GMT+0800 (China Standard Time)

When I set racket-program to something like "NA" which doesn't exist, I can reproduce the file-mode-specification error if I open some file outside of an envrc dir.

But I can't seem to reproduce 1 or 2, where I'm in an envrc dir -- exec-path is updated from that soon enough, racket is found, no error.

The timing/order seems to differ for you and me. If that's really what's going on, and depending why, then that would make me nervous that the whole idea of using envrc-mode-hook is unreliable and I should abandon this.

But for now still looking/thinking....

Karl H · Answer 11 · Thu Apr 11 2024 23:47:21 GMT+0800 (China Standard Time)

The timing/order could also be because of my recent switch to elpaca and my config being quite a mess.

I am not 100% sure that envrc-mode-hook actually fires when the environment variables get loaded by envrc. From reading the documentation, it seems like that hook fires when (envrc-mode) is called, instead of when the env-vars are loaded and Emacs' notion of those variables is updated to reflect that.

Greg Hendershott · Answer 12 · Fri Apr 12 2024 02:10:19 GMT+0800 (China Standard Time)

From my read of the docs and envrc-mode code, I believe the order to be:

The buffer visits a file.

A major mode is chosen.

The major mode hook is run. This might enable minor modes like racket-xp-mode -- which will definitely try to issue a command to a back end. If this happens to be in the hook list before envrc-mode, that would be bad.

But it's worse. You and I aren't enabling envrc-mode via a hook. Instead we call envrc-global-mode in our init files. envrc-global-mode is defined using define globalized-minor-mode, whose doc string says "When a major mode is initialized, MODE is actually turned on just after running the major mode's hook." After. If I'm reading that correctly, envrc-mode will be initialized after any minor modes enabled via the major mode hook (like racket-xp-mode). So that's even worse, it's guaranteed to be too late. 😞

So why does it seem to work, at all? In my case it was initially silently using the global "/" back end and my global racket (until I disabled that to experiment). In your case it's erroring initially, but later working with the envrc specific back end when that eventually is created. I think??

The fact that the ordering is so sensitive and intricate, makes me feel like this was a good intention, but a bad idea.

Probably .dir-locals.el is the least-worst idea (if, like you, one prefers to keep this in the file system alongside the .envrc files).

Or, some people might prefer doing the racket-back-end in their Emacs init file. (I think your preference is better, FWIW.)

TL;DR: I'm leaning toward giving up on this, and instead switching to documenting the need to do dir-locals with envrc.

Greg Hendershott · Answer 13 · Fri Apr 12 2024 02:34:53 GMT+0800 (China Standard Time)

TL;DR: I'm leaning toward giving up on this, and instead switching to documenting the need to do dir-locals with envrc.

On the issue-706 branch I just pushed a commit 6970ee7 to do this.