"panic: runtime error: slice bounds out of range" after non SS traffic

[Artem Vetrov]

Hello. After couple of days normal work, in one moment I find out that my shadowsocks client can't connect to server. I logged via ssh and see that the shadowsocks service down, because goquiet chashed. There is journalctl -xe excerpt:

Oct 03 15:00:08 vps ss-server[433]: 2018/10/03 15:00:08 +1 non SS traffic from 151.68.102.223:36605
Oct 03 15:00:08 vps ss-server[433]: panic: runtime error: slice bounds out of range
Oct 03 15:00:08 vps ss-server[433]: goroutine 351460 [running]:
Oct 03 15:00:08 vps ss-server[433]: github.com/cbeuw/GoQuiet/gqserver.ReadTillDrain(0x57b200, 0xc00000e128, 0xc00072d000, 0x5000, 0x5000, 0x0, 0x0, 0x0)
Oct 03 15:00:08 vps ss-server[433]:         /home/andy/go/src/github.com/cbeuw/GoQuiet/gqserver/util.go:54 +0x30e
Oct 03 15:00:08 vps ss-server[433]: main.(*ssPair).remoteToServer(0xc00000c420)
Oct 03 15:00:08 vps ss-server[433]:         /home/andy/go/src/github.com/cbeuw/GoQuiet/cmd/gq-server/gq-server.go:68 +0x9c
Oct 03 15:00:08 vps ss-server[433]: created by main.dispatchConnection.func2
Oct 03 15:00:08 vps ss-server[433]:         /home/andy/go/src/github.com/cbeuw/GoQuiet/cmd/gq-server/gq-server.go:117 +0xc8
Oct 03 15:00:08 vps ss-server[433]:  2018-10-03 15:00:08 ERROR: plugin service exit unexpectedly
Oct 03 15:00:08 vps systemd[1]: shadowsocks-libev-goquet.service: Main process exited, code=exited, status=255/n/a
Oct 03 15:00:08 vps systemd[1]: shadowsocks-libev-goquet.service: Unit entered failed state.
Oct 03 15:00:08 vps systemd[1]: shadowsocks-libev-goquet.service: Failed with result 'exit-code'.

shadowsocks-libev 3.2.0
goquiet 1.2.1
ubuntu 16.04.5 (2.6.32-042stab127.2 kernel)

Thank you, and sorry my bad English.

[Andy Wang]

It appears that the crash is caused by the non-SS traffic, but gqserver.ReadTillDrain shouldn't have been called at all after the "non SS traffic" has been printed.

So what may have happened is that the data sent by that non SS traffic somehow got read by ReadTillDrain on a goroutine that handles a connection from gqclient, which shouldn't have happened since they are on two separate TCP streams.

I suspect race condition. I'll do some test with the go race detector.

[Artem Vetrov]

If this may help in your investigation, there is only one request from non-SS traffic that goquiet has redirected to local web server, before crashed himself

[Andy Wang]

Judging by the log from your web server, here's a possible reconstruction of what happened:

That IP sent a well-formed ClientHello (well formed in terms of header info at least) since gqserver.ParseClientHello didn't throw an error on gq-server.go:131. Then this ClientHello went through gqserver.IsSS, which returned false and the "+1 non SS traffic" info was logged. goWeb then made a connection with your local web server and tried to send the ClientHello through. However, in the middle of sending this, gqserver crashed. This can be known by the fact that your web server only received the header of it instead of the entire message.

I still don't see how this would trigger a panic in gqserver.ReadTillDrain that was called by remoteToServer. Maybe the non SS connection was unrelated and the appearances together are just a coincidence.

If I put a validation of the buffer size before https://github.com/cbeuw/GoQuiet/blob/master/gqserver/util.go#L54 then it shouldn't crash anymore. But still I don't know what could've caused this.

I'll keep the race detector running for a while and see if it catches anything

[Artem Vetrov]

Just reporting, there is no more crashes was happened after months of server uptime.

Thank you :)