GUI symbol card randomly stop working for random symbols
dasbts opened this issue · comments
Version
v0.0.98 latest Git Pull hash (running local docker-compose build)
Description
Both the action timer (for the last tick update) and the TradingView signals randomly stop working for random symbol pairs.
I have in previous issue which was closed tried both reconnecting websocket and improvements of the current socket solutions to call the function again for ws.onclose or ws('onclose', ...).
I have also tried looking into the PubSub mechanism, applying things like PubSub.clearAllSubscriptions() as well as some try of automatically restart the bot when it happens, but haven't succeeded. I noticed there are functions such as refreshTickersClean in /binance-trading-bot/app/server-binance.js but not sure if that's the right approach.
The closest I came was that with the reconnecting websocket for some reason TradingView resumed it's updates after manually triggering this by disconnecting the internet cable. However, it might behave differently upon total internet connection is lost, compared to when waiting for any of these random GUI symbol cards to stop updating.
Since this happens randomly, it is not a viable solution to restart the bot with cronjob, as it can happen anything between minutes to a day, which is why I suggest to try this out with as many symbols as possible if trying to reproduce the bug. The outcome of this is that the bot stops completely trading the symbol that the card is related to, which is a severe flaw.
To Reproduce
Add, for example, 50 symbols actively tracked, make sure they have a grid active (could probably use paper trading to simulate this), wait for some hours or up to a day, and then look at the cards on the GUI. Some should have frozen completely, not updating anything at all, while some keep on like usual.
Expected Behaviours
The action timer (and the corresponding websocket) should hopefully restart if its connection is lost and resume updating the GUI card for that symbol pair.
Screenshots
Additional note
Let me know if there is some way to log specific symbols, or if there's anything else I can provide to fix this, as I have spent way too much time investigating it and really want to fix this. Hopefully this time, it is an ok use of the issue template.
I would suggest having a check for the "Action" timer which shows last ticker update, and if that is more than 15 minutes back, then try to re-establish the connection, maybe by removing and adding the symbol card again, or use the same functionality as is available for when one removes the symbol card (as this seem to remove the websocket), then if it was not successful, wait like 30-60 seconds and retry reconnecting to the symbol card websocket.
I think I have figured it out and implemented a solution that automatically reconnects to a symbol if it is not updated within for example one minute. Confirmed with console log and an example that had not had activity for 15 minutes.
However, it only worked for those symbols that are on the GUI at the moment. I need to figure out how to do the same in some backend file as right now I put it in CoinWrapperAction.js.
Super excited for this as it would give a lot of better uptime and trade opportunities, missed sell PnL yesterday because the ticker stopped, so I'm still holding the position.
However, it is also seemingly triggering many times for each symbol, so sometimes it doesn't get time to reconnect as it seem to take some time for that process, and it thus retriggers the reconnect before finishing up. Investigating.
95% sure I have a working solution now, last night's try was only on the front end side, now having it work from the app instead of in the public. Will look into Tradingview with same approach ASAP as I've seen same behaviour (although very rare, it usually autoreconnects already) there.
@chrisleekr @rando128 @uhliksk hope it's ok to ping you like this. I've worked on some PR before, and there was then a master and develop branch. Here I see both branches with fixes, and pushes into the master if I understand it correctly, as well as PR's on your profiles. How do you want me to propose this change? Will do some more testing first however.
This is a very important fix as the current version of the bot is randomly stopping working for random symbols.
Ok, so I got it to do the actual restarting. Had to comment out the code for dust transfers in tickers.js tho, otherwise it kept adding GUI symbol cards of BTC equivalents. However, I think it also needs to be cleared in the cache or something, as sometimes I think redlock is holding it up, so the previous values are still kept after restart of the socket. Got a lot to do with work the rest of this week, but will look into the cache, or if the problem is in some other part of the code, hopefully during this weekend.
The dust transactions part, it feels like this was making the out of memory issue somehow, as I believe it had tickers running in the background for a lot of symbols BTC equivalent, even if it required a PubSub restart of the symbol for the BTC equivalent to actually appear in the GUI.
However, it seems like it works for most parts, just that some are locked up after restart, this as a full network disconnect multiple times resulted in new data coming in, something which previously was impossible (at least for me).
I managed to clear the cache for the symbol and retried with disconnect of internet followed by a wait of 10 minutes.
I set all the way of 10 minutes away as it seems sometimes the data does not reload due to the exchange (which the tooltip says), or it just has a delay (at least according to the GUI) for like 1-2 minutes sometimes before updating. Waited about 10 minutes and started seeing the symbols restarting one by one, then all back to 100% normal.
I also added a retry mechanism to TradingView as that sometimes seem to just stop working with no retry involved, so will make sure that works as expected before pushing a PR. But please see my previous question where I @ your profile name.
EDIT: Managed to add axios-retry to keep retrying forever. Will push a PR ASAP.
hello, what's up now?
hello everyone, do you got some news?
i have similar problem. i try searching where its problem