UBWA - subscribe_to_stream bug
benniebendiksen opened this issue · comments
Solution to Issue cannot be found in the documentation or other Issues and also occurs in the latest version of this library.
- I checked the documentation and other Issues. I am using the latest version of this library.
Version of this library.
2.4.0
Hardware?
VPS or other cloud hosting
Operating System?
Linux
Python version?
Python3.12
Installed wheel files
unicorn_binance_rest_api-2.2.1.dist-info/INSTALLER
unicorn_binance_rest_api-2.2.1.dist-info/LICENSE
unicorn_binance_rest_api-2.2.1.dist-info/METADATA
unicorn_binance_rest_api-2.2.1.dist-info/RECORD
unicorn_binance_rest_api-2.2.1.dist-info/WHEEL
unicorn_binance_rest_api-2.2.1.dist-info/top_level.txt
unicorn_binance_rest_api/__init__.cpython-312-x86_64-linux-gnu.so
unicorn_binance_rest_api/enums.cpython-312-x86_64-linux-gnu.so
unicorn_binance_rest_api/exceptions.cpython-312-x86_64-linux-gnu.so
unicorn_binance_rest_api/helpers.cpython-312-x86_64-linux-gnu.so
unicorn_binance_rest_api/manager.cpython-312-x86_64-linux-gnu.soInstalled packages
# Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
aiohttp 3.9.3 pypi_0 pypi
aiosignal 1.3.1 pypi_0 pypi
aniso8601 9.0.1 pypi_0 pypi
appdirs 1.4.4 pyhd3eb1b0_0
attrs 23.1.0 py312h06a4308_0
automat 20.2.0 py_0
bcrypt 3.2.0 py312h5eee18b_1
blas 1.0 mkl
blinker 1.7.0 pypi_0 pypi
bottleneck 1.3.7 py312ha883a20_0
brotli-python 1.0.9 py312h6a678d5_7
bzip2 1.0.8 h5eee18b_5
ca-certificates 2024.3.11 h06a4308_0
certifi 2024.2.2 py312h06a4308_0
cffi 1.16.0 py312h5eee18b_0
charset-normalizer 2.0.4 pyhd3eb1b0_0
cheroot 10.0.0 pypi_0 pypi
click 8.1.7 pypi_0 pypi
colorama 0.4.6 pypi_0 pypi
constantly 23.10.4 py312h06a4308_0
cryptography 42.0.5 py312hdda0065_0
cython 3.0.9 pypi_0 pypi
dateparser 1.2.0 pypi_0 pypi
expat 2.5.0 h6a678d5_0
flask 3.0.2 pypi_0 pypi
flask-restful 0.3.10 pypi_0 pypi
frozenlist 1.4.1 pypi_0 pypi
hyperlink 21.0.0 pyhd3eb1b0_0
idna 3.4 py312h06a4308_0
incremental 22.10.0 pyhd3eb1b0_0
intel-openmp 2023.1.0 hdb19cb5_46306
itsdangerous 2.1.2 pypi_0 pypi
jaraco-functools 4.0.0 pypi_0 pypi
jinja2 3.1.3 pypi_0 pypi
ld_impl_linux-64 2.38 h1181459_1
libffi 3.4.4 h6a678d5_0
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libstdcxx-ng 11.2.0 h1234567_1
libuuid 1.41.5 h5eee18b_0
lucit-licensing-python 1.8.2 pypi_0 pypi
markupsafe 2.1.5 pypi_0 pypi
mkl 2023.1.0 h213fc3f_46344
mkl-service 2.4.0 py312h5eee18b_1
mkl_fft 1.3.8 py312h5eee18b_0
mkl_random 1.2.4 py312hdb19cb5_0
more-itertools 10.2.0 pypi_0 pypi
multidict 6.0.5 pypi_0 pypi
ncurses 6.4 h6a678d5_0
numexpr 2.8.7 py312hf827012_0
numpy 1.26.4 py312hc5e2394_0
numpy-base 1.26.4 py312h0da6c21_0
openssl 3.0.13 h7f8727e_0
pandas 2.1.1 py312h526ad5a_0 anaconda
pip 23.3.1 py312h06a4308_0
psutil 5.9.8 pypi_0 pypi
pyasn1 0.4.8 pyhd3eb1b0_0
pyasn1-modules 0.2.8 py_0
pycparser 2.21 pyhd3eb1b0_0
pycryptodome 3.20.0 pypi_0 pypi
pyopenssl 24.0.0 py312h06a4308_0
pysocks 1.7.1 py312h06a4308_0
python 3.12.2 h996f2a0_0
python-binance 1.0.19 pypi_0 pypi
python-dateutil 2.8.2 pyhd3eb1b0_0
python-tzdata 2023.3 pyhd3eb1b0_0
pytz 2023.3.post1 py312h06a4308_0
readline 8.2 h5eee18b_0
regex 2023.12.25 pypi_0 pypi
requests 2.31.0 py312h06a4308_0 anaconda
service_identity 18.1.0 pyhd3eb1b0_1
setuptools 68.2.2 py312h06a4308_0
shared-memory-dict 0.7.2 pypi_0 pypi
simplejson 3.19.2 pypi_0 pypi
six 1.16.0 pyhd3eb1b0_1
sqlite 3.41.2 h5eee18b_0
tbb 2021.8.0 hdb19cb5_0
tk 8.6.12 h1ccaba5_0
twisted 23.10.0 py312h06a4308_0
typing_extensions 4.9.0 py312h06a4308_1
tzdata 2024a h04d1e81_0
tzlocal 5.2 pypi_0 pypi
ujson 5.9.0 pypi_0 pypi
unicorn-binance-rest-api 2.2.1 pypi_0 pypi
unicorn-binance-websocket-api 2.4.0 pypi_0 pypi
unicorn-fy 1.14.1 pypi_0 pypi
urllib3 1.26.18 py312h06a4308_0
websocket-client 1.7.0 pypi_0 pypi
websockets 11.0.3 pypi_0 pypi
werkzeug 3.0.1 pypi_0 pypi
wheel 0.41.2 py312h06a4308_0
xz 5.4.6 h5eee18b_0
yarl 1.9.4 pypi_0 pypi
zlib 1.2.13 h5eee18b_0
zope 1.0 py312h06a4308_1
zope.interface 5.4.0 py312h5eee18b_0Logging output
No response
Processing method?
process_stream_data
Used endpoint?
binance.com
Issue
ubwa_stream_subscribe_issue.py.log.zip
ubwa_stream_subscribe_issue.py.zip
periodic calls into subscribe_to_stream() returns a payload value of False, and the corresponding market does not undergo the subscription. This error happens periodically with no clear pattern as to when.
Print statements at the time of the first occurrence was as follows:
UNSUBSCRIBED btttrx, market count: 702
2024-04-13 23:30:16: FAILED TO SUBSCRIBE TO wintrx STREAM!!! Adding to self.unicorn_failed_market_subscriptions ...
Len of self.unicorn_failed_market_subscriptions: 1
UNSUBSCRIBED chzbnb, market count: 701
Log file simply shows that the coin in question, wintrx, was listed in the list of subscription markets, and subsequently, that the payload value was False.
Hi Pablo!
I know the code part that is responsible for this and I want to rework it because of the Binance Websocket API. I want to send directly via loop.create_task() to reduce the latency, probably I can fix your issue as well. I will report here as soon as you can try it again.
Hi Oliver,
Please forgive my delayed response. It is the busiest time of the academic semester for me.
The issue still exists as of today (version 2.4.0). From inspecting the ubwa manager class code statically, I think the issue of the False payload value may be localized here:
If this is correct then I am curious as to why calls into this split_payload() method from the "unsubscribe" method do not result in the returned value of False (I have only observed the issue with subscribe attempts).
Can you show me how calling into subscribe_to_stream() results in sending API requests via the established websocket connection? I cannot find the link between appending to self.stream_list[stream_id]['payload']—which is what I understand subscribe_to_stream() and unsubscribe_from_stream() to be doing (in the absence of this issue)—and the addition of a task to an event loop or some other mechanism to send an API request.
In any case, if self.stream_list[stream_id]['payload'] does not get appended to from within subscribe_to_stream() whenever the issue is encountered, then doesn't that mean that no request gets sent to the Binance Websocket API?
I can try to dynamically trace the issue when I have more time, if it would help, which would be after mid-May.
Good approach from you, I will test why the False occurs at all.
I first have to integrate the option of accessing the event loop directly and sending it via websocket, which is not yet available. Currently the payload is stored on a stack and another process picks it up from the stack in the loop and then sends it (not optimal ^^) In the future I would like to have this done directly via a separate asyncio task.
A legitimate reason for a False would be if max_subscriptions_per_stream is exceeded! Have you checked this? You should then find a "The limit of X has been exceeded!" in the logs and you can retrieve the current number of subscriptions:
https://unicorn-binance-websocket-api.docs.lucit.tech/unicorn_binance_websocket_api.html#unicorn_binance_websocket_api.manager.BinanceWebSocketApiManager.get_limit_of_subscriptions_per_stream
https://unicorn-binance-websocket-api.docs.lucit.tech/unicorn_binance_websocket_api.html#unicorn_binance_websocket_api.manager.BinanceWebSocketApiManager.get_number_of_subscriptions
Another reason would be the TypeError that is caught, here you should also find a log note!
I have now equipped all the functions involved with logging at the crucial points, if you activate debug level "ERROR", you will have all the relevant information in the log before the False is returned. So we can better investigate the reason.
Please test with version 2.6.0 and keep an eye on the max subscriptions!
A legitimate reason for a
Falsewould be ifmax_subscriptions_per_streamis exceeded! Have you checked this? You should then find a "The limit of X has been exceeded!" in the logs and you can retrieve the current number of subscriptions: https://unicorn-binance-websocket-api.docs.lucit.tech/unicorn_binance_websocket_api.html#unicorn_binance_websocket_api.manager.BinanceWebSocketApiManager.get_limit_of_subscriptions_per_stream https://unicorn-binance-websocket-api.docs.lucit.tech/unicorn_binance_websocket_api.html#unicorn_binance_websocket_api.manager.BinanceWebSocketApiManager.get_number_of_subscriptionsAnother reason would be the TypeError that is caught, here you should also find a log note!
I have now equipped all the functions involved with logging at the crucial points, if you activate debug level "ERROR", you will have all the relevant information in the log before the
Falseis returned. So we can better investigate the reason.
Hi Oliver,
Thank you for looking into this. I keep an eye on the stream subscription count by printing the market count associated with the stream. Also, in the example script for this issue—ubwa_stream_subscribe_issue.py, we initialize markets to not exceed the subscription limit (I believe the value is 1024 markets), i.e., markets = markets[:subscribe_limit]. The subscribe_limit we grab from the method call indicated in your first provided web link. We then associate boolean values to these markets by sampling a uniform distribution and create the stream by subscribing to the markets associated with a value of True. We then iteratively unsubscribe from subscribed markets and subscribe to unsubscribed markets. So on average there will be 50% of the subscription number limit of markets subscribed to the stream, and never will there be more than the limit. Nevertheless, as an added measure, I will go ahead and use the get_number_of_subscriptions() method you reference with your second provided web-link. But my belief is that the problem isn't there.
The TypeError, to my understanding, used to result when the program would attempt to iterate the payload variable, thereby assuming it to be of an iterable type. Yet, as we know, that payload variable is actually a Bool at times when the bug shows up. In version 2.4.0 the TypeError stopped getting thrown because of the try-catch wrap you placed before the program attempts to iterate the payload.
As for a log note in either case, I will go ahead and run the example script until the bug appears, terminate the program, and inspect all logs in the generated log file.
Thank you for extending the logger writes around more points surrounding this problem. I'll go ahead and activate the "ERROR" level of the logger when undergoing my testing.
Good approach from you, I will test why the
Falseoccurs at all.I first have to integrate the option of accessing the event loop directly and sending it via websocket, which is not yet available. Currently the payload is stored on a stack and another process picks it up from the stack in the loop and then sends it (not optimal ^^) In the future I would like to have this done directly via a separate asyncio task.
Ah, thank you for the explanation. I agree that an immediate addition of a task to a running event loop, as a result of a subscription or unsubscription method call, would be optimal.
Please test with version 2.6.0 and keep an eye on the max subscriptions!
Yes, I will definitely do so. Please allow me to do so either Tuesday or Wednesday of the upcoming week.
The limits are different "binance.com" supports 1024 while "binance.com-futures" only supports 200: https://github.com/LUCIT-Systems-and-Development/unicorn-binance-websocket-api/wiki/Binance-websocket-endpoint-configuration-overview
UBWA knows these limits and you can use get_limit_of_subscriptions_per_stream() to query the limits for the endpoint used and get_number_of_subscriptions() to calculate the free slots. Or you can make it even easier for yourself and use get_number_of_free_subscription_slots().
It would actually be useful to throw an exception as soon as too many subscriptions are made.
It would actually be useful to throw an exception as soon as too many subscriptions are made.
Agreed. Good idea
get_limit_of_subscriptions_per_stream()to query the limits for the endpoint used
I apologize, I should've specified. I was referring to "binance.com"; I've not tested for the issue in the futures markets. I'll make sure to keep calling get_number_of_free_subscription_slots() to verify that free slots remain available at times when the payload bug occurs.
from unicorn_binance_websocket_api import BinanceWebSocketApiManager, MaximumSubscriptionsExceeded
try:
ubwa.subscribe_to_stream(stream_id=stream_id, channels=['kline_1m', 'depth20'])
except MaximumSubscriptionsExceeded as e:
print(f"ERROR: {e}")
Please let me know if you still have problems or if I can close the issue.
ubwa_stream_subscribe_issue_2.py.log
ubwa_stream_subscribe_issue_2.py.zip
Hi Oliver,
Please note the log file and its corresponding test py file above. The test file utilizes both the "MaximumSubscriptionsExceeded" Exception and also calls the "get_number_of_free_subscription_slots(stream_id)" method when the bug is encountered.
I understand now why you desired to inspect the number of subscriptions to ensure we were not exceeding the maximum subscription number when encountering the error. Here is a mismatch for you regarding this topic:
Print statements from the client side indicate that we do not exceed the subscription limit (1024 for Binance.com CEX):
...
SUBSCRIBED arpausdt, market count: 703
SUBSCRIBED trxbusd, market count: 703
SUBSCRIBED eosbusd, market count: 703
SUBSCRIBED iotxusdt, market count: 703
UNSUBSCRIBED rlcusdt, market count: 702
2024-05-16 01:01:01: FAILED TO SUBSCRIBE TO mcousdt STREAM!!! Adding to self.unicorn_failed_market_subscriptions
Num free slots: 322 ...
Yet, inspecting the log file we see that:
2024-05-16 01:00:23,629 [ERROR ] 3720 139648502327040 events: BinanceWebSocketApiManager.send_with_stream(a473242bad0d-11b3-4a3e-8141-92f4e838 - Timeout exceeded!
2024-05-16 01:00:28,655 [ERROR ] 3720 139648502327040 events: BinanceWebSocketApiManager.send_with_stream(a473242bad0d-11b3-4a3e-8141-92f4e838 - Timeout exceeded!
2024-05-16 01:00:35,954 [ERROR ] 3720 139648502327040 events: BinanceWebSocketApiManager.send_with_stream(a473242bad0d-11b3-4a3e-8141-92f4e838 - Timeout exceeded!
2024-05-16 01:00:40,976 [ERROR ] 3720 139648502327040 events: BinanceWebSocketApiManager.send_with_stream(a473242bad0d-11b3-4a3e-8141-92f4e838 - Timeout exceeded!
2024-05-16 01:00:45,999 [ERROR ] 3720 139648502327040 events: BinanceWebSocketApiManager.send_with_stream(a473242bad0d-11b3-4a3e-8141-92f4e838 - Timeout exceeded!
2024-05-16 01:01:00,376 [ERROR ] 3720 139648502327040 events: BinanceWebSocketApiSocket.start_socket(a473242bad0d-11b3-4a3e-8141-92f4e838) - Received error message: {"error":{"code":4,"msg":"Too many subscriptions"},"id":1533}
2024-05-16 01:01:01,510 [ERROR ] 3720 139648872167040 ubwa_stream_subscribe_issue_2: BinanceWebSocketApiManager.split_payload() CEX result is None!
2024-05-16 01:01:01,511 [ERROR ] 3720 139648872167040 ubwa_stream_subscribe_issue_2: BinanceWebSocketApiManager.subscribe_to_stream(a473242bad0d-11b3-4a3e-8141-92f4e838) - error_msg: Payload is None!
2024-05-16 01:01:01,874 [ERROR ] 3720 139648502327040 events: BinanceWebSocketApiManager.split_payload() CEX result is None!
2024-05-16 01:01:01,874 [ERROR ] 3720 139648502327040 events: BinanceWebSocketApiManager.subscribe_to_stream(a473242bad0d-11b3-4a3e-8141-92f4e838) - error_msg: Payload is None!
So something is happening that is causing a "Too many subscriptions" error to be received by the websocket. UBWA has no awareness of this over-subscribing as indicated by the fact that, in subscribe_to_stream(), even though the logger reports that payload is None, it never reports that the limit of 1024 subscriptions per stream has been exceeded; in other words—if self.stream_list[stream_id]['subscriptions'] > self.max_subscriptions_per_stream:—is not getting hit.
The payload value of None can be traced to the call to self.split_payload() here:
which receives a None returned value coming from here:
Since the logger reports a sequence of "BinanceWebSocketApiManager.send_with_stream(a473242bad0d-11b3-4a3e-8141-92f4e838 - Timeout exceeded!" errors before the "too many subscriptions" error, do you think that maybe self.stream_list[stream_id]['payload'] is getting overloaded from multiple calls into self.add_payload_to_stream() in the following
?
I dont think its "overloaded", i will investigate this, but give me some time
Yes of course, thank you for your time.