WS orderbook sync unable to recover after UpdateLocalCache error (Bittrex, possibly other exchanges as well)

Question

WS orderbook sync unable to recover after UpdateLocalCache error (Bittrex, possibly other exchanges as well)

TaltaM opened this issue 3 years ago · comments

TaltaM commented 3 years ago

New Issue

Context

Bittrex (and possibly other exchanges) fail to recover after a websocket orderbook update is unsuccessfully applied.

Running debian 9, with GoCryptoTrader compiled from commit 126187b.

./gocryptotrader --version
GoCryptoTrader v0.1 amd64 go1.16.7 pre-release.

I tried relaxing timeouts, job limits and max workers to mitigate the issue, accounting for possibly slow connections, and although I think this does mitigate the issue, it does not avoid it.

Expected Behavior

When a WS orderbook update for Bittrex is received that cannot be applied, the expected behavior is for the orderbook to be re-initialized (cleaning buffer, making a REST call and continuing with applying updates).

Current Behavior

Currently, if the stream is continued, the updates aren't successfully applied to the book:
exchange Bittrex websocket error - Bittrex - UpdateLocalCache error: websocket orderbook synchronisation failure for pair BTC-USDT and asset spot

Failure Information (for bugs)

Empty slices and missing updates

The first issue is that although the book is cleared after an error is observed, the code that monitors for empty books does tests for a nil slice, not for a slice with len == 0:
https://github.com/thrasher-corp/gocryptotrader/blob/master/exchanges/bittrex/bittrex_ws_orderbook.go#L148

This should help there:

@@ -145,7 +158,7 @@ func (b *Bittrex) applyBufferUpdate(pair currency.Pair) error {
        }
 
        recent, err := b.Websocket.Orderbook.GetOrderbook(pair, asset.Spot)
-       if err != nil || (recent.Asks == nil && recent.Bids == nil) {
+       if err != nil || (recent.Asks == nil && recent.Bids == nil) || (len(recent.Asks) == 0 && len(recent.Bids) == 0) {
                if b.Verbose {
                        log.Debugf(log.WebsocketMgr, "Orderbook: Fetching via REST\n")
                }

However, after manually triggering failed updates, the REST call is performed but updates aren't successfully applied (see third log snippet)

I've noted that the ID of the update is larger than the expected ID:

This appears to be because the orderbook's LastUpdateID isn't saved.

I can manually do AssignOptions after doing GetDepth on the orderbook:

diff --git a/exchanges/bittrex/bittrex_ws_orderbook.go b/exchanges/bittrex/bittrex_ws_orderbook.go
index ca4c2d99..a2a6558d 100644
--- a/exchanges/bittrex/bittrex_ws_orderbook.go
+++ b/exchanges/bittrex/bittrex_ws_orderbook.go
@@ -130,7 +144,16 @@ func (b *Bittrex) SeedLocalCacheWithOrderBook(p currency.Pair, sequence int64, o
        newOrderBook.LastUpdateID = sequence
        newOrderBook.VerifyOrderbook = b.CanVerifyOrderbook
 
-       return b.Websocket.Orderbook.LoadSnapshot(&newOrderBook)
+       err := b.Websocket.Orderbook.LoadSnapshot(&newOrderBook)
+       if err != nil {
+               return err
+       }
+       depth, err := orderbook.GetDepth(b.Name, p, asset.Spot)
+       if err != nil {
+               return err
+       }
+       depth.AssignOptions(&newOrderBook)
+       return nil
 }
 
 // applyBufferUpdate applies the buffer to the orderbook or initiates a new

I'm not sure that's the proper way, or if flushAndCleanup() should be adapted..

The residual issue is that the orderbook is empty until the REST call returned new values. I wonder if that's preferable over setting a flag that the data is stale or something similar while keeping the book populated.

But it seems like I largely solved the issue for the Bittrex exchange while writing this report, and I'll open a PR with those changes...

Binance has a similar erroneous test for empty books, but the problem doesn't manifest itself there because Binance appears to continue updating fields in the book if there are missing updates.
I had similar issues with OKEX and OKCoin International, where at some point the websocket couldn't recover.

Steps to Reproduce

I did a quick hack and modified the getticker RPC call to trigger the skipping of an update:

diff --git a/exchanges/bittrex/bittrex.go b/exchanges/bittrex/bittrex.go
index ced133b6..849bf04f 100644
--- a/exchanges/bittrex/bittrex.go
+++ b/exchanges/bittrex/bittrex.go
@@ -25,6 +25,7 @@ type Bittrex struct {
 
        obm         *orderbookManager
        tickerCache *TickerCache
+       skipcounter int
 }
 
 const (
diff --git a/exchanges/bittrex/bittrex_wrapper.go b/exchanges/bittrex/bittrex_wrapper.go
index ddb9716c..dea326d7 100644
--- a/exchanges/bittrex/bittrex_wrapper.go
+++ b/exchanges/bittrex/bittrex_wrapper.go
@@ -326,6 +326,7 @@ func (b *Bittrex) constructTicker(t TickerData, s *MarketSummaryData, pair curre
 
 // FetchTicker returns the ticker for a currency pair
 func (b *Bittrex) FetchTicker(p currency.Pair, assetType asset.Item) (*ticker.Price, error) {
+       b.skipcounter = -1
        resp, err := ticker.GetTicker(b.Name, p, assetType)
        if err != nil {
                return b.UpdateTicker(p, assetType)
diff --git a/exchanges/bittrex/bittrex_ws_orderbook.go b/exchanges/bittrex/bittrex_ws_orderbook.go
index ca4c2d99..b0d35650 100644
--- a/exchanges/bittrex/bittrex_ws_orderbook.go
+++ b/exchanges/bittrex/bittrex_ws_orderbook.go
@@ -38,6 +39,10 @@ func (b *Bittrex) setupOrderbookManager() {
 
 // ProcessUpdateOB processes the websocket orderbook update
 func (b *Bittrex) ProcessUpdateOB(pair currency.Pair, message *OrderbookUpdateMessage) error {
+       if b.skipcounter == -1 {
+               b.skipcounter++
+               return errors.New("Inserting error")
+       }
        var updateBids []orderbook.Item
        for x := range message.BidDeltas {
                updateBids = append(updateBids, orderbook.Item{

step 1: Enable the Bittrex exchange, its websocket stream and the BTC-USDT pair
step 2: Wait for an update to fail - or manually trigger
step 3: Monitor logs or the orderbook stream

Failure Logs

Log snippet 1:

[DEBUG] | 10/08/2021 20:10:48 | Bittrex websocket connection: message received: {"C":"d-7377AEED-B,0|CKpq,0|CKpr,4|Go,64DF7|ud,131B1|CKps,13E3","M":[{"H":"C3","M":"ticker","A":["q1YqrsxNys9RslJyCnHWDQ12CVHSUcpJLC4JKUpMSQ1KLEkFS
[INFO]  | 10/08/2021 20:10:48 | Bittrex websocket btcusdt SPOT: TICKER: Last 45136.10470000 Ask 45107.01000000 Bid 45091.57043110 High 46691.44459265 Low 44637.61600000 Volume 258.70253236
[DEBUG] | 10/08/2021 20:10:48 | Bittrex websocket connection: message received: {"C":"d-7377AEED-B,0|CKpq,0|CKpr,4|Go,64DF8|ud,131B1|CKps,13E3","T":1,"M":[{"H":"C3","M":"orderBook","A":["jY/LDoIwEEX/Zda1mZaWAkvlD8CVcVG0iYSHAmVB
[INFO]  | 10/08/2021 20:10:48 | Bittrex websocket btcusdt SPOT: ORDERBOOK: Bids len: 500 Amount: 49.374545 BTC. Total value: 2146787.087494005 Asks len: 500 Amount: 48.666891 BTC. Total value: 2289953.4559427532
[DEBUG] | 10/08/2021 20:10:48 | Bittrex websocket: running connection monitor cycle
[WARN]  | 10/08/2021 20:10:48 | Bittrex Received empty message
[WARN]  | 10/08/2021 20:10:48 | Bittrex websocket has been disconnected. Reason: websocket: close 1000 (normal)
[DEBUG] | 10/08/2021 20:10:48 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 10/08/2021 20:10:50 | Bittrex attempt 1 request path: https://socket-v3.bittrex.com/signalr/negotiate?connectionData=[{name:"c3"}]&clientProtocol=1.5
[DEBUG] | 10/08/2021 20:10:50 | Bittrex request type: GET
[DEBUG] | 10/08/2021 20:10:50 | HTTP status: 200 OK, Code: 200
[DEBUG] | 10/08/2021 20:10:50 | Bittrex raw response: {"Url":"/signalr","ConnectionToken":"APBvf/rgGvtr63fFufmhWaaT35qppcBBG/qiceCvmxrvlesYvCptFr2PkuDtbumr7gJgrYuAvz8ox6aHNvglppnSyRn7hHJRdsYlqfofY8aAztMh","ConnectionId":"8496f2
[DEBUG] | 10/08/2021 20:10:50 | Bittrex websocket: setting unauthenticated websocket URL: wss://socket-v3.bittrex.com/signalr/connect?clientProtocol=1.5&connectionData=%5B%7Bname%3A%22c3%22%7D%5D&connectionToken=APBvf%2FrgGvtr6
[INFO]  | 10/08/2021 20:10:50 | Bittrex Websocket connected to wss://socket-v3.bittrex.com/signalr/connect?clientProtocol=1.5&connectionData=%5B%7Bname%3A%22c3%22%7D%5D&connectionToken=APBvf%2FrgGvtr63fFufmhWaaT35qppcBBG%2Fqice
[DEBUG] | 10/08/2021 20:10:50 | Bittrex - Sending JSON message - {"H":"c3","M":"subscribe","A":[["orderbook_BTC-USDT_500","ticker_BTC-USDT","market_summary_BTC-USDT"]],"I":161655877}
[DEBUG] | 10/08/2021 20:10:50 | Bittrex websocket connection: sending message to websocket {Hub:c3 Method:subscribe Arguments:[[orderbook_BTC-USDT_500 ticker_BTC-USDT market_summary_BTC-USDT]] InvocationID:161655877}
[DEBUG] | 10/08/2021 20:10:50 | Bittrex websocket connection: message received: {"C":"d-E35EC538-B,0|C1D,0|C1E,1","S":1,"M":[]}
[DEBUG] | 10/08/2021 20:10:50 | Bittrex websocket connection: message received: {"C":"d-E35EC538-B,0|C1D,0|C1E,2|JB,94A","G":"QAiuLlC65vCULvGQk46ffgbtb4abuu3nwNkXOQ4Gl77BpirP9kcIQadMplVCDv2AhHqzBfy++lHyqwkAb1F7gmfOdV9y4moV4r9ky
[DEBUG] | 10/08/2021 20:10:50 | Bittrex websocket connection: message received: {"C":"d-E35EC538-B,0|C1D,0|C1E,3|JB,94A|W6,13F","G":"6ZsnnRz78LsXWMcorpBUV7hpvICTafXGdb4IakwTkylymamqbfPrGv9YinEltA/CoFyfWCZH5TGEW6kfUGU6kZPgw34FH7
[DEBUG] | 10/08/2021 20:10:50 | Bittrex websocket connection: message received: {"R":[{"Success":true,"ErrorCode":null},{"Success":true,"ErrorCode":null},{"Success":true,"ErrorCode":null}],"I":"161655877"}
[DEBUG] | 10/08/2021 20:10:50 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 10/08/2021 20:10:50 | Bittrex websocket connection: message received: {"C":"d-E35EC538-B,0|C1D,0|C1E,4|JB,94A|W6,13F|C1L,0","G":"9dfRgQ/DMZ67sbv8m3KCbtBSMGJxA2LZ/GoWqO9T6jSbBGuI2ba0EcETHFQB5iy+mk0s9pxItsawWgmX5FTrk9/r
[DEBUG] | 10/08/2021 20:10:50 | Bittrex websocket connection: message received: {"C":"d-E35EC538-B,0|C1D,0|C1E,4|JB,94B|W6,13F|C1L,0","M":[{"H":"C3","M":"orderBook","A":["VYy9DoIwGEXf5c61+VppajsqbwBOhqFIEwk/Ci0DIby7ddF4t5PcczYM
[ERROR] | 10/08/2021 20:10:50 | exchange Bittrex websocket error - Bittrex - UpdateLocalCache error: websocket orderbook synchronisation failure for pair BTC-USDT and asset spot
[DEBUG] | 10/08/2021 20:10:51 | Bittrex websocket connection: message received: {"C":"d-E35EC538-B,0|C1D,0|C1E,4|JB,94C|W6,13F|C1L,0","M":[{"H":"C3","M":"orderBook","A":["lZLPDoIwDIffpefZlLLBxlF9A/VkOAxdohHwD/NgjO/uNEYlhCi9tf2+
[ERROR] | 10/08/2021 20:10:51 | exchange Bittrex websocket error - Bittrex - UpdateLocalCache error: websocket orderbook synchronisation failure for pair BTC-USDT and asset spot
[DEBUG] | 10/08/2021 20:10:51 | Bittrex websocket connection: sending message []
[DEBUG] | 10/08/2021 20:10:51 | Bittrex websocket connection: message received: {"C":"d-E35EC538-B,0|C1D,0|C1E,4|JB,94D|W6,13F|C1L,0","M":[{"H":"C3","M":"orderBook","A":["ldLLbkIhEAbgd5k1JcNlGDhL6xtoV00XWElqvLR6cGGM715O07TH5JAg
[ERROR] | 10/08/2021 20:10:51 | exchange Bittrex websocket error - Bittrex - UpdateLocalCache error: websocket orderbook synchronisation failure for pair BTC-USDT and asset spot

(etc)

Log snippet 2:

[DEBUG] | 11/08/2021 14:30:40 | Bittrex websocket connection: message received: {"C":"d-88ACFB50-F,0|DGNu,0|DGNv,4|Db,3E253|Dm:,9C7A|C6zY,363","M":[{"H":"C3","M":"orderBook","A":["VY27DoMgAEX/5c6UIAUVRusfaKfGAStJjY9WwcEY/r10ado
[INFO]  | 11/08/2021 14:30:40 | Bittrex websocket btcusdt SPOT: ORDERBOOK: Bids len: 500 Amount: 44.046231 BTC. Total value: 1941410.447475651 Asks len: 500 Amount: 53.079398 BTC. Total value: 2539653.0460294476
[DEBUG] | 11/08/2021 14:30:42 | Bittrex websocket: running connection monitor cycle
[WARN]  | 11/08/2021 14:30:43 | Bittrex Received empty message
[WARN]  | 11/08/2021 14:30:43 | Bittrex websocket has been disconnected. Reason: websocket: close 1006 (abnormal closure): unexpected EOF
[DEBUG] | 11/08/2021 14:30:43 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 11/08/2021 14:30:44 | Bittrex attempt 1 request path: https://socket-v3.bittrex.com/signalr/negotiate?connectionData=[{name:"c3"}]&clientProtocol=1.5
[DEBUG] | 11/08/2021 14:30:44 | Bittrex request type: GET
[DEBUG] | 11/08/2021 14:30:44 | HTTP status: 200 OK, Code: 200
[DEBUG] | 11/08/2021 14:30:44 | Bittrex raw response: {"Url":"/signalr","ConnectionToken":"qGRFo132hwuaboff1MmOiCPj7+yUv+m/YoPJd6SgNk10IKB7ZeoyIQYkAzRW9X+DsBVY/IvOsKC/oIL6PVbT0Sak9aSgAOq/zoG8HljXuShn8wml","ConnectionId":"0254c7
[DEBUG] | 11/08/2021 14:30:44 | Bittrex websocket: setting unauthenticated websocket URL: wss://socket-v3.bittrex.com/signalr/connect?clientProtocol=1.5&connectionData=%5B%7Bname%3A%22c3%22%7D%5D&connectionToken=qGRFo132hwuabof
[INFO]  | 11/08/2021 14:30:44 | Bittrex Websocket connected to wss://socket-v3.bittrex.com/signalr/connect?clientProtocol=1.5&connectionData=%5B%7Bname%3A%22c3%22%7D%5D&connectionToken=qGRFo132hwuaboff1MmOiCPj7%2ByUv%2Bm%2FYoPJ
[DEBUG] | 11/08/2021 14:30:44 | Bittrex - Sending JSON message - {"H":"c3","M":"subscribe","A":[["orderbook_BTC-USDT_500","ticker_BTC-USDT","market_summary_BTC-USDT"]],"I":184684342}
[DEBUG] | 11/08/2021 14:30:44 | Bittrex websocket connection: sending message to websocket {Hub:c3 Method:subscribe Arguments:[[orderbook_BTC-USDT_500 ticker_BTC-USDT market_summary_BTC-USDT]] InvocationID:184684342}
[DEBUG] | 11/08/2021 14:30:44 | Bittrex websocket connection: message received: {"C":"d-E35EC538-B,0|DP0K,0|DP0L,1","S":1,"M":[]}
[DEBUG] | 11/08/2021 14:30:44 | Bittrex websocket connection: message received: {"C":"d-E35EC538-B,0|DP0K,0|DP0L,2|JB,3ED41","G":"KgwJ4kXWE5BbhuFKq0gTeheDonwh4XqVM304Ws/US/RFKJvabNOEWxPNNIYcPgD4xsALJl2xEPAhhzvJIITGtaXWcDB6ZIg6S
[DEBUG] | 11/08/2021 14:30:44 | Bittrex websocket connection: message received: {"C":"d-E35EC538-B,0|DP0K,0|DP0L,3|JB,3ED41|DDPS,6EC","G":"itsIB/9QayhDobYUczsKigtRph6LTP1lWpzB7+PNsQ5zxCsZ1rrMRd8dKJInTbyonrOAwcspsPj+goJmRGHoKjwv
[DEBUG] | 11/08/2021 14:30:44 | Bittrex websocket connection: message received: {"R":[{"Success":true,"ErrorCode":null},{"Success":true,"ErrorCode":null},{"Success":true,"ErrorCode":null}],"I":"184684342"}
[DEBUG] | 11/08/2021 14:30:44 | Bittrex websocket connection: message received: {"C":"d-E35EC538-B,0|DP0K,0|DP0L,4|JB,3ED41|DDPS,6EC|DOcW,3A","G":"QXjT0bgmQi/PbP3z3bejkgWmAwkayjk8cspZQHZ9tD/CsSWhja8fTBGAfiLCQ/3B2wTkHT0+RDY9mpmu
[DEBUG] | 11/08/2021 14:30:44 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 11/08/2021 14:30:44 | Bittrex websocket connection: message received: {"C":"d-E35EC538-B,0|DP0K,0|DP0L,4|JB,3ED42|DDPS,6EC|DOcW,3A","M":[{"H":"C3","M":"orderBook","A":["VY27DoIwFIbf5Z8rOcX2aDsqbwBOhqFIEwkXBcpACO9uJ6P7
[ERROR] | 11/08/2021 14:30:44 | exchange Bittrex websocket error - Bittrex - UpdateLocalCache error: websocket orderbook synchronisation failure for pair BTC-USDT and asset spot
[DEBUG] | 11/08/2021 14:30:44 | Bittrex websocket connection: message received: {"C":"d-E35EC538-B,0|DP0K,0|DP0L,4|JB,3ED43|DDPS,6EC|DOcW,3A","M":[{"H":"C3","M":"orderBook","A":["XY29DoMgAITf5WZqAJEWRusbaKfGAStJjT+tgoMxvHtxadLe
[ERROR] | 11/08/2021 14:30:44 | exchange Bittrex websocket error - Bittrex - UpdateLocalCache error: websocket orderbook synchronisation failure for pair BTC-USDT and asset spot

(etc)

After fixing the book presence test and adding one line of debugging info:

diff --git a/exchanges/bittrex/bittrex_ws_orderbook.go b/exchanges/bittrex/bittrex_ws_orderbook.go
index ca4c2d99..c71369a1 100644
--- a/exchanges/bittrex/bittrex_ws_orderbook.go
+++ b/exchanges/bittrex/bittrex_ws_orderbook.go
@@ -194,6 +203,7 @@ func (b *Bittrex) processJob(p currency.Pair) error {
        // new update to initiate this.
        err = b.applyBufferUpdate(p)
        if err != nil {
+               log.Errorf(log.WebsocketMgr, "%s applying buffer update error %v", b.Name, err)
                b.flushAndCleanup(p)
                return err
        }

[DEBUG] | 12/08/2021 12:55:09 | Bittrex websocket connection: message received: {"C":"d-6CD3CA60-B,0|B6O4,0|B6O5,4|Ej,25AB3|BYs,70D4|B0Ip,182","M":[{"H":"C3","M":"orderBook","A":["fY7BDoIwDIbfpee5rKwF4ai8AXgyHIYs0QgoMA6E8O5OY2L
U6H9p0nzf387QmP5sXTY15aWGBDb5drXL0hwEVPbqjpCwUgIG2422PVhIMCRmFlCeqtTWzgyQ7GfoRtO6k5t8g5JqrR7xHb1x3gEixSgjfu6XQoAZzj99xE+fSbP3w9gngkW8Gz/A9V8wilGSRtRI9AVKfz5grellcIABypj8xPsPxXID"]}]}
[ERROR] | 12/08/2021 12:55:09 | exchange Bittrex websocket error - Bittrex - UpdateLocalCache error: BTC-USDT spot processing update error: Inserting error
[DEBUG] | 12/08/2021 12:55:10 | Bittrex websocket connection: message received: {"C":"d-6CD3CA60-B,0|B6O4,0|B6O5,4|Ej,25AB4|BYs,70D4|B0Ip,182","M":[{"H":"C3","M":"orderBook","A":["dY5BDoIwEEXvMuvaTKVTSpfKDcCVcVGkiURAgbIghLtbozE
hxFn+vPcyMzS2vzufTU3xqMHAIT/uTlmaA4PSPf0NDCEyGFw3uvbqwAgliRSDoipTV3s7gDnP0I229ZWfQgE5CoHvC43e+uCApChRXO+VCrOAha2NFag1j1QSBzDagPwTXqUlkuAxffflwsAO93+vkdz4RKg46p+/vAA="]}]}
[DEBUG] | 12/08/2021 12:55:10 | Orderbook: Fetching via REST
[DEBUG] | 12/08/2021 12:55:10 | Bittrex attempt 1 request path: https://api.bittrex.com/v3/markets/BTC-USDT/orderbook?depth=500
[DEBUG] | 12/08/2021 12:55:10 | Bittrex request type: GET
[DEBUG] | 12/08/2021 12:55:10 | HTTP status: 200 OK, Code: 200
[DEBUG] | 12/08/2021 12:55:10 | Bittrex raw response: {"bid":[{"quantity":"0.06552885","rate":"45406.39639999"},{"quantity":"0.06603120","rate":"45401.91758446"},{"quantity":"0.06603505","rate":"45399.22469897"},{"quantity":"0.
01100000","rate":"45396.82660001"},{"quantity":"0.250200 **<SNAP REST RESPONSE>**

[ERROR] | 12/08/2021 12:55:10 | Bittrex applying buffer update error initial websocket orderbook sync failure for pair BTC-USDT and asset spot
[INFO]  | 12/08/2021 12:55:10 | Bittrex websocket btcusdt SPOT: ORDERBOOK: Bids len: 500 Amount: 46.069856 BTC. Total value: 2019172.180867939 Asks len: 500 Amount: 50.754773 BTC. Total value: 2404067.0364411827
[ERROR] | 12/08/2021 12:55:10 | Bittrex processing websocket orderbook error initial websocket orderbook sync failure for pair BTC-USDT and asset spot
[DEBUG] | 12/08/2021 12:55:10 | Bittrex websocket connection: message received: {"C":"d-6CD3CA60-B,0|B6O4,0|B6O5,4|Ej,25AB5|BYs,70D4|B0Ip,182","M":[{"H":"C3","M":"orderBook","A":["fdBNbgIxDAXgu3idWnZiZ5JZAjeArhCLAJGK+CtMWCA0d2f
UqmopTbN+37PjG+zTeZvL9LpfHnfQwmg2fnmdTmZgYJ3fyxu0SmSgy6dLPqwytOxFtTGw3KwneVdSB+38BqdLOpRNuQ4NNNBzKkMURJ0EVEsfD3rzT5AFxVeCyF8V30JIGRutCWIrLvrmhyCOgkRVQRI1cnwQIaDY4Bw5hn5hIHXbyq/RytOOqiTIoTbR0V/CI9XEQ1CGs1QPpmR/V3vvPLqG4qdY9Hc="]
}]}
[DEBUG] | 12/08/2021 12:55:10 | Orderbook: Fetching via REST
[DEBUG] | 12/08/2021 12:55:10 | Bittrex websocket connection: message received: {"C":"d-6CD3CA60-B,0|B6O4,0|B6O5,4|Ej,25AB6|BYs,70D4|B0Ip,182","M":[{"H":"C3","M":"orderBook","A":["jZDNCsIwEITfZc8xbJJNTHpU30A9iYfUBiz98afxINJ3N4i
CWgrd4+58w8w+oPHXKsT1vclPNWSw2Cxn2/VqAwyKcI5HyDQigy5cbqE9BMiEIa0tg7wsVqGOvoNs94DLzbexjPfkgFwSviZ5XH1MDJBW5DjN3/ue/RHvwy+hDCc3Rlhhh4TQnMQI8SVE4YgjThFay0lapVAJ6PcMfFeNtP4OQjJVkpKUcXIYHR0qg9r8EMZyZ9JrhZtYVivLpfl02PdP"]}]}
[DEBUG] | 12/08/2021 12:55:10 | Bittrex websocket connection: message received: {"C":"d-6CD3CA60-B,0|B6O4,0|B6O5,4|Ej,25AB7|BYs,70D4|B0Ip,182","M":[{"H":"C3","M":"orderBook","A":["bYzLDoIwEEX/Zda1mT4pLJU/AFeGRZEmEh4KlAUh/LuNC5X
obG4mOees0NmxcT5buvLeQgLH/HQ4Z2kOBCr38DdIFCKByQ2z668OEqalUjGBsq5S13o7QXIpCNipeb8rDLPtfe2XEMRQGq0PJkgleUS1NFoiImxkD1LUGnlsxM4QjMbIRKQF/zG+QYWaosHX/UlzFkcB+RiKM8mpUVEYo2Ertic="]}]}
[DEBUG] | 12/08/2021 12:55:10 | Bittrex websocket connection: message received: {"C":"d-6CD3CA60-B,0|B6O4,0|B6O5,4|Ej,25AB8|BYs,70D4|B0Ip,182","M":[{"H":"C3","M":"orderBook","A":["VYy9DoIwAITf5ebatPTH2lF5A3AyDEWaSPhRaBkI4d2ti8b
lLne5+zYMbu58LNahfvawOJeXw7XISxA0/hUfsIoxguCnxY93D8u1VDo1ddvkvo8uwN4qAhe6b9wwLW6MbVwTkFGmNRNJEnJ2MSEglcw4PSnDhDAaO/l//IYq4zKjRh2TfYbV/gY="]}]}
[DEBUG] | 12/08/2021 12:55:10 | Bittrex websocket connection: message received: {"C":"d-6CD3CA60-B,0|B6O4,0|B6O5,4|Ej,25AB8|BYs,70D5|B0Ip,182","M":[{"H":"C3","M":"ticker","A":["q1YqrsxNys9RslJyCnHWDQ12CVHSUcpJLC4JKUpMSQ1KLEkFSp
mYmhiY6RlbmhlbAgFQQVJmCi6pxOJshJSRqZ6RoakBCCjVAgA="]}]}
[INFO]  | 12/08/2021 12:55:10 | Bittrex websocket btcusdt SPOT: TICKER: Last 45406.39639999 Ask 45425.21500000 Bid 45406.39639999 High 46731.92870769 Low 44849.40250699 Volume 213.62631567
[DEBUG] | 12/08/2021 12:55:11 | Bittrex websocket connection: message received: {"C":"d-6CD3CA60-B,0|B6O4,0|B6O5,4|Ej,25AB9|BYs,70D5|B0Ip,182","M":[{"H":"C3","M":"orderBook","A":["VY29DoIwGEXf5c61+YptlY7IG4CTYSjSRMKPAmUgpO9uB2P
0zuecu2Owc+d8sQ31s4dBVl4O1yIvwdC4l3/AKCKGxU2rG+8ORmiptGCo2yZ3vbcLzG3HtNrRt36LBYrqbH1EIdUx1fycaE1EAoH9g5xEIiNx+jFIpJLTZwgVg12671EV3g=="]}]}
[DEBUG] | 12/08/2021 12:55:11 | Bittrex attempt 1 request path: https://api.bittrex.com/v3/markets/BTC-USDT/orderbook?depth=500
[DEBUG] | 12/08/2021 12:55:11 | Bittrex request type: GET
[DEBUG] | 12/08/2021 12:55:11 | HTTP status: 200 OK, Code: 200
[DEBUG] | 12/08/2021 12:55:11 | Bittrex raw response: {"bid":[{"quantity":"0.06552885","rate":"45406 **<SNAP REST RESPONSE>**

Scott · Answer 1 · Fri Aug 13 2021 06:35:18 GMT+0800 (China Standard Time)

Thank you for such a detailed issue report! Recovering websocket orderbooks has been an issue to address for some time and I appreciate the investigation.

TaltaM · Answer 2 · Fri Aug 13 2021 18:58:02 GMT+0800 (China Standard Time)

More than welcome!

I'll post another snippet below, although I'm aware that you guys are aware: when switching from WS to REST after a timeout, it doesn't switch back.
(Sadly I have no clue why the timeout occurs in the first place, literally)

[DEBUG] | 13/08/2021 01:19:38 | Bittrex websocket connection: message received: {"C":"d-6CD3CA60-B,0|CLp2,0|CLp3,4|Ej,4EF0C|BYs,E81D|CLp6,1AA3","M":[{"H":"C3","M":"orderBook","A":["VYzNCoMwAIPfJedOWtuK7XH6BrrT8FBnYeLPpq0Hkb77OgZjC+SQkC8HJrMO1lf71D5GaJzr4nSpyhoEnX36O7SklMDZZbPzzUJzzjMqCNq+K+3ojYO+NgTGDd94YNnM7Hu/x0Man
1bjIwmRs1wkacoU41IikP9hQqOy6B9CqfTTv4XQhBc="]}]}
[INFO]  | 13/08/2021 01:19:38 | Bittrex websocket btcusdt SPOT: ORDERBOOK: Bids len: 500 Amount: 47.802302 BTC. Total value: 2049556.7540323136 Asks len: 500 Amount: 41.432739 BTC. Total value: 1914514.1314561588
[DEBUG] | 13/08/2021 01:19:38 | Bittrex websocket connection: sending message []
[DEBUG] | 13/08/2021 01:19:39 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:19:41 | Portfolio manager: Successfully updated address balance for BTC address(es) [1JCe8z4jJVNXSjohjM4i9Hh813dLCNx2Sy 3Nxwenay9Z8Lc9JBiywExpnEFiLp6Afp8v]
[ERROR] | 13/08/2021 01:19:41 | PortfolioWatcher error portfolio item does not exist for currency LTC
[DEBUG] | 13/08/2021 01:19:41 | Portfolio manager: Successfully updated address balance for ETH address(es) [0xb794f5ea0ba39494ce839613fffba74279579268]
[DEBUG] | 13/08/2021 01:19:41 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:19:43 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:19:45 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:19:47 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:19:49 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:19:50 | Bittrex websocket connection: sending message []
[DEBUG] | 13/08/2021 01:19:51 | Bittrex websocket connection: sending message []
[DEBUG] | 13/08/2021 01:19:51 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:19:53 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:19:55 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:19:57 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:19:59 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:20:01 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:20:03 | Bittrex websocket connection: sending message []
[DEBUG] | 13/08/2021 01:20:03 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:20:04 | Bittrex websocket connection: sending message []
[DEBUG] | 13/08/2021 01:20:05 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:20:07 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:20:09 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:20:11 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:20:13 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:20:15 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:20:16 | Bittrex websocket connection: sending message []
[DEBUG] | 13/08/2021 01:20:17 | Bittrex websocket connection: sending message []
[DEBUG] | 13/08/2021 01:20:17 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:20:19 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:20:21 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:20:23 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:20:25 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:20:27 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:20:29 | Bittrex websocket connection: sending message []
[DEBUG] | 13/08/2021 01:20:29 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:20:30 | Bittrex websocket connection: sending message []
[DEBUG] | 13/08/2021 01:20:31 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:20:33 | Bittrex websocket: running connection monitor cycle
[WARN]  | 13/08/2021 01:20:34 | Bittrex btcusdt SPOT: No ticker update after 1m0s, switching from websocket to rest
[DEBUG] | 13/08/2021 01:20:34 | Bittrex attempt 1 request path: https://api.bittrex.com/v3/markets/BTC-USDT/ticker
[DEBUG] | 13/08/2021 01:20:34 | Bittrex request type: GET
[DEBUG] | 13/08/2021 01:20:34 | HTTP status: 200 OK, Code: 200
[DEBUG] | 13/08/2021 01:20:34 | Bittrex raw response: {"symbol":"BTC-USDT","lastTradeRate":"44216.76721535","bidRate":"44225.97000000","askRate":"44238.71771338"}
[DEBUG] | 13/08/2021 01:20:35 | Bittrex attempt 1 request path: https://api.bittrex.com/v3/markets/BTC-USDT/summary
[DEBUG] | 13/08/2021 01:20:35 | Bittrex request type: GET
[DEBUG] | 13/08/2021 01:20:35 | HTTP status: 200 OK, Code: 200
[DEBUG] | 13/08/2021 01:20:35 | Bittrex raw response: {"symbol":"BTC-USDT","high":"46193.09046632","low":"43798.74700000","volume":"211.24777103","quoteVolume":"9457616.40193775","percentChange":"-3.34","updatedAt":"2021-08-12T23:20:34.81Z"}
[INFO]  | 13/08/2021 01:20:35 | Bittrex REST btcusdt SPOT: TICKER: Last 44216.76721535 Ask 44238.71771338 Bid 44225.97000000 High 46193.09046632 Low 43798.74700000 Volume 211.24777103
[DEBUG] | 13/08/2021 01:20:35 | Bittrex websocket: running connection monitor cycle
[DEBUG] | 13/08/2021 01:20:37 | Bittrex websocket: running connection monitor cycle
[WARN]  | 13/08/2021 01:20:38 | Bittrex btcusdt SPOT: No orderbook update after 1m0s, switching from websocket to rest
[DEBUG] | 13/08/2021 01:20:38 | Bittrex attempt 1 request path: https://api.bittrex.com/v3/markets/BTC-USDT/orderbook?depth=500
[DEBUG] | 13/08/2021 01:20:38 | Bittrex request type: GET
[DEBUG] | 13/08/2021 01:20:38 | HTTP status: 200 OK, Code: 200
[DEBUG] | 13/08/2021 01:20:38 | Bittrex raw response: {"bid":[{"quantity":"0.03000000","rate":"44217.68000000"},{"quantity":"0.06783133","rate":"44204.47107477"},{"quantity":"0.06783808","rate":"44199.71111537"},{"quantity":"0.50838000","rate":"44196.78500000"},{"quantity":"0.25020000","rate":"44191.28800000"},{"quan
tity":"0.11304209","rate":"44190.93865293"},{"quantity":"0.06600000","rate":"44177.12000001"},{"quantity":"0.01100000","rate":"44153.33929208"},{"quantity": **<SNAP REST RESPONSE>**

After this, no switching back.

Adrian Gallagher · Answer 3 · Wed Aug 18 2021 09:42:59 GMT+0800 (China Standard Time)

Thanks for the great report and #747 PR, tested master by manually closing the connection and seeing the processing errors. Then tested again with the PR on #747 and can see that it resyncs correctly.