bug: Zombie users (reproduceable)

Question

bug: Zombie users (reproduceable)

opened this issue 4 years ago · comments

Deleted user commented 4 years ago

Expected Behavior

User will be kicked when the moderator types

/kick user

Actual Behavior

When using a certain client and exiting, the user lingers and cannot be kicked or banned until restarting the server.

Steps to reproduce behavior

Get an android phone.
Download the app termius from the Google Play Store.
Open the app and click on continue without account.
Go to hosts, click on new host, and enter the details such as your username. Make sure to enter ssh.chat as the IP address.
Once you've successfully ssh into the ssh.chat, completely close the app but leave it running in the background. (This can be done by just swiping the app away). Now let your phone idle on battery saving mode for a long time (I'd say around 45 minutes).
Go back to the app, and swipe right to reveal a menu. Go to terminal, which will show you a list of active terminals.
Finally, close the terminal by pressing the close button. The user should linger and is unable to be kicked or banned.

Additional Comments

This bug is extremely hard to reproduce intentionally and requires some patience.

I noticed that the client for the un-kickable users was SSH-2.0-libssh2_1.9.0_DEV (maybe this can be produced with other apps that support this type of client?).

Android version: 7
Battery Saving: On
SSH Client: Termius

Andrey Petrov · Answer 1 · Tue Aug 11 2020 00:41:29 GMT+0800 (China Standard Time)

cc @voldyman you were looking at this before, any interest in investigating? :)

Akshay Shekher · Answer 2 · Tue Aug 11 2020 00:48:24 GMT+0800 (China Standard Time)

This is interesting, what happens when you try to kick the user after they've reached this state?

Andrey Petrov · Answer 3 · Tue Aug 11 2020 00:49:10 GMT+0800 (China Standard Time)

@voldyman The usual zombie behaviour we saw before,

12:01  onelegend_: I found a possible bug
12:02  [🍮 shazow] which was...
12:03  onelegend_: try kicing onelegend
12:03  onelegend_: or Guest2458
12:03  [🍮 shazow] /kick onelegend
12:03   * onelegend was kicked by 🍮 shazow.
12:03  [🍮 shazow] /whois onelegend
12:03  -> name: onelegend
 > ip: [snip]
 > fingerprint: (no public key)
 > client: SSH-2.0-libssh2_1.9.0_DEV
 > joined: 2.6 days ago

Andrey Petrov · Answer 4 · Tue Aug 11 2020 00:50:48 GMT+0800 (China Standard Time)

Maybe a good way is to setup a server with a debugger attached/tracing, and try connecting and dropping from termius.

Akshay Shekher · Answer 5 · Tue Aug 11 2020 00:51:56 GMT+0800 (China Standard Time)

yeah, that would be a good starting point.

Andrey Petrov · Answer 6 · Sat Mar 27 2021 00:03:17 GMT+0800 (China Standard Time)

Having trouble reproducing locally T_T

$ ssh-anon -p 2022 localhost
Warning: Permanently added '[localhost]:2022' (RSA) to the list of known hosts.
 * shazow joined. (Connected: 1)
 * Guest1 joined. (Connected: 2)
 * Guest1 left. (After 2 seconds)
 * shazow left. (After 3 minutes)
 * shazow joined. (Connected: 1)
shazow: test
 * Guest3 joined. (Connected: 2)
[Guest3] hi
[Guest3] ping shazow
 * shazow left. (After 65 seconds)
 * u0_a241 joined. (Connected: 2)
u0_a241: hi
[Guest3] /whois u0_a241
-> name: u0_a241
 > fingerprint: (no public key)
 > client: SSH-2.0-OpenSSH_8.0
 > joined: 4 seconds ago
 * shazow joined. (Connected: 3)
[Guest3] /whois shazow
-> name: shazow
 > fingerprint: (no public key)
 > client: SSH-2.0-libssh2_1.9.0_DEV
 > joined: 4 seconds ago
[Guest3] /whois shazow
-> name: shazow
 > fingerprint: (no public key)
 > client: SSH-2.0-libssh2_1.9.0_DEV
 > joined: 99 seconds ago
 * shazow left. (After 9 minutes)
 * u0_a241 left. (After 12 minutes)
[Guest3] /whois shazow
-> Err: user not found

Deleted user · Answer 7 · Sat Mar 27 2021 01:48:11 GMT+0800 (China Standard Time)

cc @shazow Wait until you can't physically type in the terminal to close it.

Andrey Petrov · Answer 8 · Sat Mar 27 2021 21:24:36 GMT+0800 (China Standard Time)

@epicgoldenlife I haven't been able to get it to that state, any other tips? I find the user disconnects within a few minutes of swiping away the app, at least on Android 11.

Deleted user · Answer 9 · Sun Mar 28 2021 10:50:08 GMT+0800 (China Standard Time)

I'd say instead of swiping away the app, try just pressing the home key and locking the screen. I know this isn't part of the original procedure to reproduce this bug, but I think android may have been aggressively killing the app in your case.

Andrey Petrov · Answer 10 · Mon Mar 29 2021 21:30:18 GMT+0800 (China Standard Time)

Tried that too, still can't reproduce it. :( Can you try running the latest master ssh-chat locally and connecting to it from your phone and see if you can reproduce it locally?

Deleted user · Answer 11 · Thu Apr 01 2021 10:17:40 GMT+0800 (China Standard Time)

I also noticed that when I connect to sshchat and then hibernate my windows device, the user stays connected for a few minutes, and I'm even able to resume the session like nothing happened when the computer is booted up again.

Andrey Petrov · Answer 12 · Thu Apr 01 2021 10:37:25 GMT+0800 (China Standard Time)

It's not related, that's just an artifact of the ssh timeout procedure and the ability to resume tcp sockets until they timeout.

Andrey Petrov · Answer 13 · Fri Apr 02 2021 05:21:16 GMT+0800 (China Standard Time)

I spun up a dedicated server to populate with zombies, and we got some success. Here are the results.

Debug logs: (It seems zombies1-5 were indeed zombies, though zombie5 might have zombified in a special way?)

...
2021-03-31 21:26:51.071 DEBUG [:48516] Joined: zombie
[chat/message] 2021/03/31 22:27:35 Write failed to zombie, closing: write tcp :4242->:48516: write: connection reset by peer
2021-04-01 09:51:56.193 DEBUG [:51278] Joined: zombie2
2021-04-01 09:53:26.017 DEBUG [:51295] Joined: zombie3
2021-04-01 09:53:53.882 DEBUG [:51296] Joined: Guest12
2021-04-01 10:29:54.867 DEBUG [:51345] Joined: Guest13
[chat/message] 2021/04/01 10:29:54 Write failed to zombie2, closing: EOF
[chat/message] 2021/04/01 10:29:54 Write failed to zombie3, closing: EOF
[chat/message] 2021/04/01 10:29:54 Write failed to zombie4, closing: EOF
2021-04-01 10:37:01.495 DEBUG [:51379] Joined: Guest14
2021-04-01 10:40:26.700 DEBUG [:51345] Leaving: zombie5
2021-04-01 10:40:27.399 DEBUG [:51379] Leaving: shaz0w
2021-04-01 10:47:16.192 DEBUG [:51410] Joined: zombie5
2021-04-01 11:41:50.235 DEBUG [:34086] Leaving: onelegend
[chat/message] 2021/04/01 11:41:50 Write failed to zombie5, closing: write tcp :4242->:51410: write: connection reset by peer
[sshd] 2021/04/01 12:30:35 [:57341] Failed to handshake: ssh: overflow reading version string
2021-04-01 12:44:29.330 DEBUG [:28524] Joined: onelegend
2021-04-01 12:44:40.158 DEBUG [:56582] Joined: shazow
2021-04-01 12:45:27.678 DEBUG [:28524] Leaving: onelegend
2021-04-01 12:45:36.147 DEBUG [:56582] Leaving: shazow
[sshd] 2021/04/01 12:45:55 [:56592] Failed to handshake: ssh: overflow reading version string
[sshd] 2021/04/01 12:45:56 [:56590] Failed to handshake: ssh: overflow reading version string
[sshd] 2021/04/01 12:46:16 [:56594] Failed to handshake: read tcp :4242->:56594: i/o timeout

pprof goroutine dump:

goroutine profile: total 36
5 @ 0x43a8e5 0x4059fa 0x405755 0x69ffdd 0x69fa57 0x69f6f5 0x470f21
#	0x69ffdc	golang.org/x/crypto/ssh.(*mux).handleChannelOpen+0x23c	/home/shazow/local/go/pkg/mod/golang.org/x/crypto@v0.0.0-20210322153248-0c34fe9e7dc2/ssh/mux.go:293
#	0x69fa56	golang.org/x/crypto/ssh.(*mux).onePacket+0x216		/home/shazow/local/go/pkg/mod/golang.org/x/crypto@v0.0.0-20210322153248-0c34fe9e7dc2/ssh/mux.go:231
#	0x69f6f4	golang.org/x/crypto/ssh.(*mux).loop+0x34		/home/shazow/local/go/pkg/mod/golang.org/x/crypto@v0.0.0-20210322153248-0c34fe9e7dc2/ssh/mux.go:190

5 @ 0x43a8e5 0x4068cf 0x40654b 0x6807c5 0x6d8aa5 0x470f21
#	0x6807c4	golang.org/x/crypto/ssh.(*channel).SendRequest+0x284	/home/shazow/local/go/pkg/mod/golang.org/x/crypto@v0.0.0-20210322153248-0c34fe9e7dc2/ssh/channel.go:591
#	0x6d8aa4	github.com/shazow/ssh-chat/sshd.NewTerminal.func1+0x104	/home/shazow/local/go/src/github.com/shazow/ssh-chat/sshd/terminal.go:122

5 @ 0x43a8e5 0x4068cf 0x40654b 0x68655e 0x470f21
#	0x68655d	golang.org/x/crypto/ssh.DiscardRequests+0x3d	/home/shazow/local/go/pkg/mod/golang.org/x/crypto@v0.0.0-20210322153248-0c34fe9e7dc2/ssh/connection.go:80

5 @ 0x43a8e5 0x4068cf 0x40654b 0x6d7c05 0x470f21
#	0x6d7c04	github.com/shazow/ssh-chat/sshd.(*Terminal).listen+0x64	/home/shazow/local/go/src/github.com/shazow/ssh-chat/sshd/terminal.go:181

5 @ 0x43a8e5 0x46d2f8 0x46d2ce 0x478879 0x67b3bc 0x67ef7c 0x6801fb 0x6d4edd 0x6d4b16 0x6ecca5 0x6d8989 0x470f21
#	0x46d2cd	sync.runtime_notifyListWait+0xcd					/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/runtime/sema.go:513
#	0x478878	sync.(*Cond).Wait+0x98							/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/sync/cond.go:56
#	0x67b3bb	golang.org/x/crypto/ssh.(*buffer).Read+0x1db				/home/shazow/local/go/pkg/mod/golang.org/x/crypto@v0.0.0-20210322153248-0c34fe9e7dc2/ssh/buffer.go:94
#	0x67ef7b	golang.org/x/crypto/ssh.(*channel).ReadExtended+0x1fb			/home/shazow/local/go/pkg/mod/golang.org/x/crypto@v0.0.0-20210322153248-0c34fe9e7dc2/ssh/channel.go:351
#	0x6801fa	golang.org/x/crypto/ssh.(*channel).Read+0x5a				/home/shazow/local/go/pkg/mod/golang.org/x/crypto@v0.0.0-20210322153248-0c34fe9e7dc2/ssh/channel.go:528
#	0x6d4edc	github.com/shazow/ssh-chat/sshd/terminal.(*Terminal).readLine+0x33c	/home/shazow/local/go/src/github.com/shazow/ssh-chat/sshd/terminal/terminal.go:833
#	0x6d4b15	github.com/shazow/ssh-chat/sshd/terminal.(*Terminal).ReadLine+0x75	/home/shazow/local/go/src/github.com/shazow/ssh-chat/sshd/terminal/terminal.go:763
#	0x6ecca4	github.com/shazow/ssh-chat.(*Host).Connect+0x544			/home/shazow/local/go/src/github.com/shazow/ssh-chat/host.go:187
#	0x6d8988	github.com/shazow/ssh-chat/sshd.(*SSHListener).Serve.func1+0x148	/home/shazow/local/go/src/github.com/shazow/ssh-chat/sshd/net.go:72

2 @ 0x43a8e5 0x4059fa 0x405755 0x687265 0x470f21
#	0x687264	golang.org/x/crypto/ssh.(*handshakeTransport).readLoop+0xc4	/home/shazow/local/go/pkg/mod/golang.org/x/crypto@v0.0.0-20210322153248-0c34fe9e7dc2/ssh/handshake.go:207

2 @ 0x43a8e5 0x44b897 0x687638 0x470f21
#	0x687637	golang.org/x/crypto/ssh.(*handshakeTransport).kexLoop+0xd7	/home/shazow/local/go/pkg/mod/golang.org/x/crypto@v0.0.0-20210322153248-0c34fe9e7dc2/ssh/handshake.go:268

1 @ 0x40c5b4 0x46d865 0x65bce5 0x470f21
#	0x46d864	os/signal.signal_recv+0xa4	/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/runtime/sigqueue.go:168
#	0x65bce4	os/signal.loop+0x24		/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/os/signal/signal_unix.go:23

1 @ 0x43a8e5 0x4068cf 0x40650b 0x72a1b6 0x43a4b6 0x470f21
#	0x72a1b5	main.main+0xbf5		/home/shazow/local/go/src/github.com/shazow/ssh-chat/cmd/ssh-chat/cmd.go:242
#	0x43a4b5	runtime.main+0x255	/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/runtime/proc.go:225

1 @ 0x43a8e5 0x4068cf 0x40654b 0x6e462f 0x470f21
#	0x6e462e	github.com/shazow/ssh-chat/chat.(*Room).Serve+0x8e	/home/shazow/local/go/src/github.com/shazow/ssh-chat/chat/room.go:125

1 @ 0x43a8e5 0x432f3b 0x46b675 0x4d3b25 0x4d5232 0x4d5214 0x56b4e5 0x57e712 0x57d6a5 0x64aa25 0x64a73a 0x72b01c 0x72af62 0x470f21
#	0x46b674	internal/poll.runtime_pollWait+0x54		/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/runtime/netpoll.go:222
#	0x4d3b24	internal/poll.(*pollDesc).wait+0x44		/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/internal/poll/fd_poll_runtime.go:87
#	0x4d5231	internal/poll.(*pollDesc).waitRead+0x211	/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/internal/poll/fd_poll_runtime.go:92
#	0x4d5213	internal/poll.(*FD).Accept+0x1f3		/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/internal/poll/fd_unix.go:401
#	0x56b4e4	net.(*netFD).accept+0x44			/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/net/fd_unix.go:172
#	0x57e711	net.(*TCPListener).accept+0x31			/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/net/tcpsock_posix.go:139
#	0x57d6a4	net.(*TCPListener).Accept+0x64			/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/net/tcpsock.go:261
#	0x64aa24	net/http.(*Server).Serve+0x284			/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/net/http/server.go:2981
#	0x64a739	net/http.(*Server).ListenAndServe+0xb9		/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/net/http/server.go:2910
#	0x72b01b	net/http.ListenAndServe+0xdb			/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/net/http/server.go:3164
#	0x72af61	main.main.func1+0x21				/home/shazow/local/go/src/github.com/shazow/ssh-chat/cmd/ssh-chat/cmd.go:81

1 @ 0x43a8e5 0x432f3b 0x46b675 0x4d3b25 0x4d5232 0x4d5214 0x56b4e5 0x57e712 0x57d6a5 0x6d69b3 0x6edbd3 0x470f21
#	0x46b674	internal/poll.runtime_pollWait+0x54				/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/runtime/netpoll.go:222
#	0x4d3b24	internal/poll.(*pollDesc).wait+0x44				/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/internal/poll/fd_poll_runtime.go:87
#	0x4d5231	internal/poll.(*pollDesc).waitRead+0x211			/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/internal/poll/fd_poll_runtime.go:92
#	0x4d5213	internal/poll.(*FD).Accept+0x1f3				/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/internal/poll/fd_unix.go:401
#	0x56b4e4	net.(*netFD).accept+0x44					/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/net/fd_unix.go:172
#	0x57e711	net.(*TCPListener).accept+0x31					/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/net/tcpsock_posix.go:139
#	0x57d6a4	net.(*TCPListener).Accept+0x64					/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/net/tcpsock.go:261
#	0x6d69b2	github.com/shazow/ssh-chat/sshd.(*SSHListener).Serve+0x92	/home/shazow/local/go/src/github.com/shazow/ssh-chat/sshd/net.go:57
#	0x6edbd2	github.com/shazow/ssh-chat.(*Host).Serve+0x72			/home/shazow/local/go/src/github.com/shazow/ssh-chat/host.go:241

1 @ 0x46b27d 0x71862e 0x718405 0x714f92 0x726865 0x7280b7 0x647844 0x6496cd 0x64a643 0x64658d 0x470f21
#	0x46b27c	runtime/pprof.runtime_goroutineProfileWithLabels+0x5c	/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/runtime/mprof.go:716
#	0x71862d	runtime/pprof.writeRuntimeProfile+0xcd			/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/runtime/pprof/pprof.go:724
#	0x718404	runtime/pprof.writeGoroutine+0xa4			/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/runtime/pprof/pprof.go:684
#	0x714f91	runtime/pprof.(*Profile).WriteTo+0x3f1			/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/runtime/pprof/pprof.go:331
#	0x726864	net/http/pprof.handler.ServeHTTP+0x384			/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/net/http/pprof/pprof.go:253
#	0x7280b6	net/http/pprof.Index+0x8d6				/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/net/http/pprof/pprof.go:371
#	0x647843	net/http.HandlerFunc.ServeHTTP+0x43			/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/net/http/server.go:2069
#	0x6496cc	net/http.(*ServeMux).ServeHTTP+0x1ac			/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/net/http/server.go:2448
#	0x64a642	net/http.serverHandler.ServeHTTP+0xa2			/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/net/http/server.go:2887
#	0x64658c	net/http.(*conn).serve+0x8cc				/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/net/http/server.go:1952

1 @ 0x63fd41 0x470f21
#	0x63fd40	net/http.(*connReader).backgroundRead+0x0	/nix/store/8an9pj9rd4izn7bjkr5nzrnw07zrdjz6-go-1.16.2/share/go/src/net/http/server.go:691