Editor lag/freezing in insert mode with rust-analyzer on macOS
inodentry opened this issue · comments
Everything works normally, until it doesn't. I type my Rust code, get autocompletions, highlights, all the LSP features work fine. ... For a few minutes.
After a while, my editor starts to lag/freeze in insert mode. As I type, the characters appear on the screen a few seconds after the keypresses. It becomes almost impossible to type, because my input appears with such a huge delay. Navigating around and selecting text, in normal mode, is still fast and responsive, so Kakoune itself isn't broken.
LSP-related autocompletion candidates are gone / no longer appear. Compiler warnings, code diagnostics, highlights, and such, stop refreshing. It all stays frozen in the same state as it was at the moment the input lag started. All LSP-related functionality stops working, LSP commands do nothing.
I can recover from this issue with :lsp-restart. Restarting LSP brings everything back up and it starts to work again (after waiting for rust-analyzer having to process my entire project again). I continue working for a few minutes, and the issue reappears. Rinse and repeat.
The issue is very annoying. I like my editor setup and can work productively ... but every 5-10 minutes or so, it becomes unusable and I need to run :lsp-restart to revive it.
Software Environment
OS: macOS Monterey 12.6.2
I successfully use Kakoune + kak-lsp + rust-analyzer on Linux, with the same configs and the same projects, and it works great. This issue seems to only happen on macOS.
Terminal: I have tried both Alacritty and the macOS Terminal.app, with and without tmux, and the issue happens always, so it is not due to the terminal.
I have had this issue for months, and finally decided to report it. It is really annoying.
During this time, I have regularly updated my Kakoune, kak-lsp, and rust-analyzer.
I build my rust-analyzer server binary from the release branch in their git repo.
I install and update kak-lsp using :plug-install plugin management from kakoune.
I have tried both the latest released version of Kakoune, as well as various recent git commits. I'm currently using Kakoune from git and updating it regularly.
If there are any relevant things from my config files that you'd like to know, let me know and I will provide them. :)
I have never used other LSP servers besides rust-analyzer, so I don't know how they behave. I don't really know if my issue is a bug in kak-lsp or a bug in rust-analyzer. I decided to report it here, because it feels most appropriate. Rust-analyzer is a popular project with lots of macOS users on other editors, so I suspect it is more likely for the bug to be related to my setup with kak-lsp, hence why I'm reporting it here.
Thanks for reporting.
When it happens, can you run set-option window debug profile, then provoke the slowness and post the contents of the *debug* buffer? This should show what exactly is slowing Kakone down. The editor is single-threaded so we must never overwhelm it with commands.
Note that you can delete the *debug* buffer, it will be recreated for the next entry.
All LSP-related functionality stops working, LSP commands do nothing.
not even lsp-capabilities?
Can you share the kak-lsp-specific part of your configuration, are you using semantic tokens? Does it also happen without any opt-in lsp options/commands?
If you can share, what's the project?
Also, can you add -vvvv --log /tmp/kak-lsp.log to your lsp_cmd and share the log up to where LSP breaks
Okay, so yesterday I did some more investigation, and I had the hypothesis that the slowness might be because of the automatic insert-mode behavior (like signature help) getting overwhelmed by rust-analyzer being too slow/unresponsive. And I think I might be correct.
I turned off all "auto in insert mode" behaviors, and the issue stopped happening!
Now I tried set-option window debug profile when the slowdown happened, and indeed I see things like this in *debug*:
shell execution took 1034775 us (spawn: 1291, wait: 1033460)
command execute-keys took 1036765 us
command evaluate-commands took 1036947 us
command try took 1048103 us
shell execution took 10581 us (spawn: 2631, wait: 7815)
command nop took 2 us
command lsp-signature-help-request took 11397 us
command evaluate-commands took 11485 us
command lsp-did-change-and-then took 1059762 us
command lsp-signature-help took 1059816 us
Another thing that confirms my suspicion is that sometimes if i wait a sufficiently long time (maybe 30+ seconds), I can see how the editor eventually "catches up" with all the activity: all the signature help info around where I typed quickly flashes on the screen and goes away.
I can then do :lsp-auto-signature-help-disable, and the editor stops being slow. No need for :lsp-restart.
not even
lsp-capabilities?
Yes, after I type a bunch with the editor being slow, all LSP commands become unresponsive. :lsp-capabilities gives no output. I now suspect that kak-lsp is just blocked waiting on RA.
If you can share, what's the project?
I mostly work with the Bevy open-source game engine and my own projects based on it. These are all pretty heavy/complex Rust projects, which is why I thought maybe RA was getting overwhelmed.
So should I report an issue to rust-analyzer? Maybe it's on them to fix their unresponsiveness?
Though, maybekak-lsp could also better handle slow servers and not slow down the editor somehow? That might be beneficial for everyone.
Actually, never mind, it happened again, even after I disabled signature help.
I see this in the debug log:
shell execution took 1033247 us (spawn: 1097, wait: 1032131)
command execute-keys took 1035140 us
command evaluate-commands took 1035320 us
command try took 1043979 us
command nop took 1 us
command evaluate-commands took 105 us
command lsp-did-change-and-then took 1044328 us
command lsp-did-change took 1044417 us
I will try to reproduce later and gather all the logs as you requested, incl. the lsp_cmd thing. I will post more info later, when I get to it. :) It might take me some time to figure out how to do it all and reproduce the slowness in a new kakoune session. The issue takes some time to manifest and I have not found a way to deliberately trigger it on-demand.
lsp-did-change should never take 1 second.
All it does it pipe the current buffer to a kak-lsp --request process, which forwards it to the kak-lsp controller process.
If the problem is that some macOS security feature slows down kak-lsp, then we could work around this by making it async (by reverting 5940c09 for example, I'll prepare a patch later) so it doesn't wait for the kak-lsp --request process to finish sending the request.
But that's just a band-aid; hopefully we can find out the source of slowness. Probably the kak-lsp logs don't give enough timing information to see what exactly is slow. Maybe macOS has useful tools, I think there's a profiler called sample
I can't explain why the kak-lsp --request process takes so long. Does it also affect empty (invalid) requests like
time echo | kak-lsp -s the_session --request
Or does it only happen with full did-change requests (for example for src/lib.rs)?
time printf %s "
session = \"the_session\"
buffile = \"$PWD/src/lib.rs\"
filetype = \"rust\"
version = 0
method = \"textDocument/didChange\"
hook = true
[params]
draft = \"\"\"
$(cat src/lib.rs)
\"\"\"
" | kak-lsp -s the_session --request
Maybe run those in a loop, something like
while timeout 0.5 echo kak-lsp -s the_session --request
do
sleep 1
doneIf the timeout is ever hit, then we can add logging to src/main.rs, mostly
the matches.is_present("request") code path (which is fairly short) to
see what exactly is slow.
I have a suspicion that this might be some weird ThinkDifferent™ thing about macOS itself. 😆
I did some googling about what could cause slow process spawning or data piping on macOS, and I found some weirdly troubling things. For example, it is possible that macOS might be randomly hijacking system calls and phoning home to Apple when spawning processes (source, source). Could explain the unpredictable nature of what I am experiencing. My editor is fine most of the time, until it seemingly-randomly becomes slow.
Interestingly, today I decided to disconnect from the internet and work offline. My editor still started lagging at some point. I just reconnected to the internet, to post this comment, and just noticed it seems to have recovered???? (or it might be because of LSP timeout and it having restarted itself...)
I don't know, maybe I am going crazy.
No, regardless of internet connection, my editor is getting really bugged.
Over the past few months (since I got my MacBookPro) I was only coding occasionally, so I didn't notice how bad the issue was. Recently, I started to work on code every day, and now I see it gets really bad. My editor is completely unusable, I see nothing while typing, and even after exiting to normal mode, I need to press Ctrl-C to issue an interrupt, to cause the editor to refresh.
I suspect there might be a lot more going on than just the lsp-did-change thing.
I will restart my editor now, with all the logging options you mentioned in previous replies.
Below workaround should make the experience acceptable again.
I tested with an artificial 1 second sleep in kak-lsp --request.
Of course completions etc. arrive a bit later but at least it doesn't block the UI.
We don't wait for any other kak-lsp --request process. We'll probably end up committing some form of this but need to work around races (5940c09).
In future, we'll probably skip spawning a process entirely (#553) which would also help Windows.
diff --git a/rc/lsp.kak b/rc/lsp.kak
index 17a53486..ca50d954 100644
--- a/rc/lsp.kak
+++ b/rc/lsp.kak
@@ -280,11 +280,15 @@ define-command -hidden lsp-did-change-and-then -params 1 -docstring %{
echo "fail"
fi
}
+ declare-option -hidden str lsp_callback "evaluate-commands -client %val{client} %arg{1}"
set-option buffer lsp_timestamp %val{timestamp}
evaluate-commands -save-regs '|' %{
set-register '|' %{
+# dump stdin synchronously
# append a . to the end, otherwise the subshell strips trailing newlines
lsp_draft=$(cat; printf '.')
+# and process it asynchronously
+(
# replace \ with \\
# " with \"
# <tab> with \t
@@ -302,10 +306,14 @@ hook = true
draft = \"\"\"
${lsp_draft}\"\"\"
" | eval "${kak_opt_lsp_cmd} --request"
+printf %s\\n "${kak_opt_lsp_callback}" | kak -p "${kak_session}"
+) > /dev/null 2>&1 < /dev/null &
}
execute-keys -draft '%<a-|><ret>'
- }}
- evaluate-commands %arg{1}
+ }
+ } catch %{
+ evaluate-commands %arg{1}
+ }
}
define-command -hidden lsp-completion -docstring "Request completions for the main cursor position" %{Unfortunately, Kakoune doesn't offer IPC beyond short-lived shell process or echo -to-file.
We need to start a kak-lsp process for every request, because they server might not be up.
If we were guaranteed that the server is up, then we could directly write to the socket.
Anyway, since we currently always need a shell process (to access the buffer contents if nothing else) this wouldn't make a difference on sane platforms.
The workaround of running kak-lsp in the background should be good enough.
I got it working in the wip branch, will commit soon.
I applied the patch from above, and can confirm that it is "working".
As in, yes, LSP still hangs sometimes, but it does not slow down the editor. I can just keep typing, and then restart LSP later at my own convenience, when I notice that I am no longer getting completions and things aren't refreshing.
Shame that macOS is doing something insane. The workaround at least makes the editor usable. Thanks! :)
Maybe we should keep the issue open, in case better ideas come up in the future, for how to better solve this?
might be a rust-analyzer issue. Can you post a log with -vvvv log level where it's slow? We can show that to rust-analyzer folks so they can determine if kak-lsp and/or macOS is doing something unreasonable.
The log file could be fairly large but it should compress well.
My educated guess is that rust-analyzer might be slow because we're not cancelling textDocument/completion requests.
The wip branch adds cancellation and should work already (but it fails a test, will fix soon).
Not cancelling requests might have something to do with it. I would suppose that kak-lsp should cancel its requests to be "well-behaved", but it wasn't implemented yet?
(No rush ... thank you so much for the work you are doing! ❤️ Kakoune and kak-lsp are a great editor environment!)
Though, if RA was hung because it was "busy" doing work, I'd expect to see CPU usage. However, there is no CPU usage from the RA process when my editor stops updating.
fully-async requests and cancellation of stale requests has been merged, so kak-lsp should work better now.
Maybe there's a problem left in rust-analyzer..