Using kitty terminal the `kitty +kitten ssh machine@ip` causes the shell to freeze during an `asciinema rec` session when asking for password input

[Samy Lahfa]

Describe the bug
Using https://github.com/kovidgoyal/kitty as a terminal, running asciinema rec test.cast and then kitty +kitten ssh, if a password prompt (see the custom password prompt) :

is asked by the remote ssh server, the shell freezes up.

To be clear, what we see during an asciinema record session is that :

• Running kitty +kitten ssh [e-mail], the shell freezes up.
• Only running kill -9 on the asciinema process allows getting back a working shell, CTRL+C does not work, nor any other signal for that matter.

To Reproduce
Steps to reproduce the behavior:

1. Launch kitty terminal
2. asciinema rec test.cast
3. kitty +kitten ssh [e-mail]
4. See a/b cases
   4a. If a password input is asked, the shell will freeze, this happens probably because kitty uses a custom way to handle password input for ssh (read additional context).
   4b. If no password prompt is asked, the ssh access works all fine, and there is no freeze, the recording is all good as well.

Expected behavior
In case a password is asked, the shell should not freeze, but the password input should be asked by kitty and then ssh login should be made, so the user can record their session.
Versions:

• OS: ArchLinux Linux 6.2.10-arch1-1
• asciinema cli: 2.2.0

Additional context
This issue seems very specific to kitty terminal emulator, and more specifically to the use of https://sw.kovidgoyal.net/kitty/kittens/ssh/
I suppose the issue is due to a conflict between how the kitty native ssh password input, askpass handles the terminal.

According to this documentation part about askpass

Control the program SSH uses to ask for passwords or confirmation of host keys etc. The default is to use kitty’s native askpass, unless the SSH_ASKPASS environment variable is set. Set this option to ssh to not interfere with the normal ssh askpass mechanism at all, which typically means that ssh will prompt at the terminal. Set it to native to always use kitty’s native, built-in askpass implementation. Note that not using the kitty askpass implementation means that SSH might need to use the terminal before the connection is established, so the kitten cannot use the terminal to send data without an extra roundtrip, adding to initial connection latency.

Setting SSH_ASKPASS=ssh and trying to reproduce the bug fails, which further shows that the bug is due to a specific interaction between asciinema and kitty's native askpass.

Here is the process tree (when the shell is frozen :

On the long line beginning by ssh -t, we can see a script being eval that contains the copyright from the maintainer of Kitty.

This piece of code may or may not contain the call to askpass, I suppose.

[Samy Lahfa]

Additional context 2, here is the code from Kitty that is called for askpass I believe : https://github.com/kovidgoyal/kitty/blob/9a0068e3188d8aca9a5a041365b81cc26f27c8b8/kittens/ssh/askpass.go#L37

[Marcin Kulik]

Thanks for comprehensive description of the problem 👌

I wonder if other terminal recording tools have similar issue when running under kitty/ssh. Are you able to check with script command or termrec?

[Samy Lahfa]

Let me give it a try soon and report on it as soon as possible !

[Samy Lahfa]

termrec

There is the same issue, but timing seems of essence indeed, waiting only a few seconds and then sending signals CTLR+C/CTRL+D/CTRL+Z seems to have crashed termrec somehow, there has been a lot of base64 (probably from kitty's custom way of sending the terminfo to the ssh machine) that was spit out inside the terminal ending with KITTY_DATA_END, trying to wait 30 sec to 1 min before sending out any signals, it seems the shell is frozen and unresponsive to any signals just like with asciinema only using kill -9 termrec unfreezes the shell and provokes the spit out of lots of base64 lines in the terminal input.

• process tree :
  

• trying to replay the recording with termplay shows this :
  
  The termrec play never ends by itself, it seems that playing it, unless I press CTRL+C.
  I'm not sure if the base64 is sensible or not, I've hidden most of it just in case.
  The terminal seems to have some borked some tty settings I suppose after playing that recording, a reset is needed so that ls's output isn't usual, it's not using 4 spaced tabs/spaces, fonts are all good, it's just spacing/formatting of output of usual commands that seems broken.

---

script

• Same behavior as termrec, it freezes the shell indefinitely just like asciinema, killing the it spits out base64 lines in the terminal ending with KITTY_DATA_END.
• See process tree before killing script :
  
• tty settings seems to be borked the same way as for termrec as well,

asciinema

The same behavior is exhibited, tty settings are somewhat modified, killing is obligatory and shell is frozen until asciinema is killed.

• Replaying the asciinema record showed this :
  

I wonder about what if I kill instead the kitten process, what would happen in that case 🤔

EDIT 2: asciinema leaves processes that hang the shell exit, termrec and script do not.

[Samy Lahfa]

Just noticed one thing, killing asciinema using SIGTERM shows askpass's prompt !

EDIT: Actually, I killed all the kitten process before doing that, then send a SIGTERM to one of the asciinema process, the more in depth in the process tree, this showed askpass prompt.

Trying to do the same without killing the kitten processes doesn't show the same behavior, the main asciinema does not budge with a SIGTERM, only a SIGKILL (=> spits back base64 into the terminal)

[Samy Lahfa]

So finally, what would you recommend in order to try and dive deeper at the root of this issue, is it worth going that far ?

Or should I just settle for a workaround such as setting SSH_ASKPASS=ssh as an env variable, or establishing the connection with a kitty shell, not running any recording, entering the password (then kitty should remember it for as long as it's running, I believe?), so I could open an ssh connection in the shell with the recording of the session.

[Marcin Kulik]

Given this problem is not specific to asciinema (as you found similar thing happens with termrec, script) I'd say the problem lies more on the kitty side. I'm not quite sure what's happening so I don't have a workaround idea at the moment.

I'm not saying asciinema is bug free in this case. In fact it's not impossible that script and termrec have the same bug. However, I find it rather unlikely that 3 separate codebases would have the same problem. Especially that for example script has been out there for over 40 years - if it had a bug like that it would most likely surface already in other terminal emulators.

[Marcin Kulik]

For completeness you can also test with ttyrec, which records in the same way as asciinema, termrec, script.

[Marcin Kulik]

From the other issue:

I don't understand also why asciinema create 3 processes, forking 2 more, when most tools only create one ?

I'll reply here since it's not related to kitty:

There's main process where copying of data between recorded process and TTY happens (the code linked above).
There's another process dedicated for file writing, in order to not block process->TTY data copying on disk I/O.
And another one for notifier (which handles triggering desktop notification in certain situations), also to not block data copying by notifications.

[Marcin Kulik]

This should be fixed in 2.3.0.

[Samy Lahfa]

I'll test and then close the issue once I tested it, thanks !