Giters

thrasher-corp / gocryptotrader

A cryptocurrency trading bot and framework supporting multiple exchanges written in Golang.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

`Websocket.Shutdown()` leaks resources

ydm opened this issue · comments

commented

Few issues with Websocket.Shutdown():

  1. Websocket.ToRoutine channel is never closed.
  2. During Websocket.Shutdown() the ShutdownC channel is recreated (~websocket.go:398), which endangers all <-ws.ShutdownC calls.
  3. After Websocket.Shutdown() there is a number of leaked gouroutines:
(dlv) gr
Thread 128647 at /usr/lib/go/src/runtime/sys_linux_amd64.s:580
Goroutine 147:
	Runtime: /usr/lib/go/src/runtime/proc.go:337 runtime.gopark (0x44add5)
	User: /usr/lib/go/src/runtime/sema.go:71 sync.runtime_SemacquireMutex (0x481227)
	Go: /home/d/go/src/github.com/ydm/gocryptotrader/exchanges/stream/websocket.go:294 github.com/thrasher-corp/gocryptotrader/exchanges/stream.(*Websocket).connectionMonitor (0xc7f5db)
	Start: /home/d/go/src/github.com/ydm/gocryptotrader/exchanges/stream/websocket.go:294 github.com/thrasher-corp/gocryptotrader/exchanges/stream.(*Websocket).connectionMonitor.func1 (0xc883e0)

(dlv) bt
0  0x000000000044add5 in runtime.gopark
   at /usr/lib/go/src/runtime/proc.go:337
1  0x000000000044ae93 in runtime.goparkunlock
   at /usr/lib/go/src/runtime/proc.go:342
2  0x000000000045dac8 in runtime.semacquire1
   at /usr/lib/go/src/runtime/sema.go:144
3  0x0000000000481227 in sync.runtime_SemacquireMutex
   at /usr/lib/go/src/runtime/sema.go:71
4  0x00000000004b5045 in sync.(*Mutex).lockSlow
   at /usr/lib/go/src/sync/mutex.go:138
5  0x00000000004b4d76 in sync.(*Mutex).Lock
   at /usr/lib/go/src/sync/mutex.go:81
6  0x0000000000c7e715 in github.com/thrasher-corp/gocryptotrader/exchanges/stream.(*Websocket).Connect
   at /home/d/go/src/github.com/ydm/gocryptotrader/exchanges/stream/websocket.go:176
7  0x0000000000c88685 in github.com/thrasher-corp/gocryptotrader/exchanges/stream.(*Websocket).connectionMonitor.func1
   at /home/d/go/src/github.com/ydm/gocryptotrader/exchanges/stream/websocket.go:338
8  0x0000000000484fe1 in runtime.goexit
   at /usr/lib/go/src/runtime/asm_amd64.s:1371

(dlv) gr
Thread 128647 at /usr/lib/go/src/runtime/sys_linux_amd64.s:580
Goroutine 163:
	Runtime: /usr/lib/go/src/runtime/proc.go:337 runtime.gopark (0x44add5)
	User: /home/d/go/src/github.com/ydm/gocryptotrader/exchanges/orderbook/node.go:99 github.com/thrasher-corp/gocryptotrader/exchanges/orderbook.(*stack).cleaner (0xacb1d2)
	Go: /home/d/go/src/github.com/ydm/gocryptotrader/exchanges/orderbook/node.go:39 github.com/thrasher-corp/gocryptotrader/exchanges/orderbook.newStack (0xacac2e)
	Start: /home/d/go/src/github.com/ydm/gocryptotrader/exchanges/orderbook/node.go:96 github.com/thrasher-corp/gocryptotrader/exchanges/orderbook.(*stack).cleaner (0xacb160)
(dlv) bt
0  0x000000000044add5 in runtime.gopark
   at /usr/lib/go/src/runtime/proc.go:337
1  0x000000000040f6c6 in runtime.chanrecv
   at /usr/lib/go/src/runtime/chan.go:576
2  0x000000000040f42b in runtime.chanrecv2
   at /usr/lib/go/src/runtime/chan.go:444
3  0x0000000000acb1d2 in github.com/thrasher-corp/gocryptotrader/exchanges/orderbook.(*stack).cleaner
   at /home/d/go/src/github.com/ydm/gocryptotrader/exchanges/orderbook/node.go:99
4  0x0000000000484fe1 in runtime.goexit
   at /usr/lib/go/src/runtime/asm_amd64.s:1371

Goroutine 199:
	Runtime: /usr/lib/go/src/runtime/proc.go:337 runtime.gopark (0x44add5)
	User: /usr/lib/go/src/runtime/sema.go:56 sync.runtime_Semacquire (0x481105)
	Go: /home/d/go/src/github.com/ydm/gocryptotrader/exchanges/stream/websocket.go:477 github.com/thrasher-corp/gocryptotrader/exchanges/stream.(*Websocket).trafficMonitor (0xc808e7)
	Start: /home/d/go/src/github.com/ydm/gocryptotrader/exchanges/stream/websocket.go:477 github.com/thrasher-corp/gocryptotrader/exchanges/stream.(*Websocket).trafficMonitor.func1 (0xc88f40)
(dlv) bt
0  0x000000000044add5 in runtime.gopark
   at /usr/lib/go/src/runtime/proc.go:337
1  0x000000000044ae93 in runtime.goparkunlock
   at /usr/lib/go/src/runtime/proc.go:342
2  0x000000000045dac8 in runtime.semacquire1
   at /usr/lib/go/src/runtime/sema.go:144
3  0x0000000000481105 in sync.runtime_Semacquire
   at /usr/lib/go/src/runtime/sema.go:56
4  0x00000000004b7a14 in sync.(*WaitGroup).Wait
   at /usr/lib/go/src/sync/waitgroup.go:130
5  0x0000000000c7fb65 in github.com/thrasher-corp/gocryptotrader/exchanges/stream.(*Websocket).Shutdown
   at /home/d/go/src/github.com/ydm/gocryptotrader/exchanges/stream/websocket.go:397
6  0x0000000000c89288 in github.com/thrasher-corp/gocryptotrader/exchanges/stream.(*Websocket).trafficMonitor.func1
   at /home/d/go/src/github.com/ydm/gocryptotrader/exchanges/stream/websocket.go:510
7  0x0000000000484fe1 in runtime.goexit
   at /usr/lib/go/src/runtime/asm_amd64.s:1371

(dlv) gr
Thread 128647 at /usr/lib/go/src/runtime/sys_linux_amd64.s:580
Goroutine 210:
	Runtime: /usr/lib/go/src/runtime/proc.go:337 runtime.gopark (0x44add5)
	User: /home/d/go/src/github.com/ydm/gocryptotrader/exchanges/stream/websocket.go:526 github.com/thrasher-corp/gocryptotrader/exchanges/stream.(*Websocket).trafficMonitor.func1.1 (0xc88f25)
	Go: /home/d/go/src/github.com/ydm/gocryptotrader/exchanges/stream/websocket.go:524 github.com/thrasher-corp/gocryptotrader/exchanges/stream.(*Websocket).trafficMonitor.func1 (0xc8959c)
	Start: /home/d/go/src/github.com/ydm/gocryptotrader/exchanges/stream/websocket.go:524 github.com/thrasher-corp/gocryptotrader/exchanges/stream.(*Websocket).trafficMonitor.func1.1 (0xc88ee0)
(dlv) bt
0  0x000000000044add5 in runtime.gopark
   at /usr/lib/go/src/runtime/proc.go:337
1  0x000000000040eb37 in runtime.chansend
   at /usr/lib/go/src/runtime/chan.go:257
2  0x000000000040e915 in runtime.chansend1
   at /usr/lib/go/src/runtime/chan.go:143
3  0x0000000000c88f25 in github.com/thrasher-corp/gocryptotrader/exchanges/stream.(*Websocket).trafficMonitor.func1.1
   at /home/d/go/src/github.com/ydm/gocryptotrader/exchanges/stream/websocket.go:526
4  0x0000000000484fe1 in runtime.goexit
   at /usr/lib/go/src/runtime/asm_amd64.s:1371

I think every long running software should be very careful with resource management.

I don't know what's on purpose and what's not, but it would be great if you come up with a plan on how to handle that and take action towards improving (and hopefully simplifying) websockets.

commented

Thanks for reporting the above @ydm! An improper shutdown can be triggered by disabling an exchanges websocket connection via gctcli. When tested on Binance, it attempts to shutdown but then gets stuck:

[DEBUG] | WEBSOCKET | 18/08/2021 10:21:02 | Binance websocket: running connection monitor cycle
[DEBUG] | WEBSOCKET | 18/08/2021 10:21:02 | Binance websocket: connectionMonitor - websocket disabled, shutting down
[DEBUG] | WEBSOCKET | 18/08/2021 10:21:02 | Binance websocket: shutting down websocket

The internal state still has the connection as enabled