[Binance] api/v1/depth Too Many Requests
anovv opened this issue · comments
Describe the bug
When I try to get l2 for a large number of pairs on Binance (>200), my IP gets rate limited.
As I understand, every time a feed is initialised it makes a snapshot request, hence for 200 pairs we get 200 simultaneous requests to depth endpoint. Is there any kind of throttling mechanism to avoid this? Shouldn't this be the same issue for other exchanges as well?
What is the best way to go here? Do I need to run multiple instances of cryptostore on different servers to bypass this? Or should I initialise different instances of cryptostore with timeouts on the same machine? Ideally I'd expect this kind of functionality to be incapsulated in cryptostore
In general, could you share best practices to use cryptostore to fetch large amounts of data (i.e. hundreds of pairs from multiple exchanges?
pretty sure this has been fixed in master on github, are you using the github latest for cryptostore and cryptofeed?
I'm using latest master. If you don't mind, could you explain what's the algorithm to bypass this problem (it will help me to better understand how cryptostore works)? Or show PR with the fix?
Could it be a problem that I'm debugging on the same machine and every time I relaunch cryptostore I send a new set of requests and get limited?
ok, I misunderstood the issue. I can subscribe to all 213 pairs and not get rate limited, so I cant reproduce this. What is the error trace?
Trace: https://pastebin.com/Z1Cqgpdi
Config: https://pastebin.com/tBnnfgZ9
subscribing to ~240 pairs
Hi, I also get this error when subscribing about 250 pairs.
Currently running latest version.
Only happens if I use CryptoStore, however if I run the example ZMQ in CryptoFeed is does not happen.
2021-03-26 13:20:57,748 : ERROR : BINANCE-a67937: encountered an exception, reconnecting
Traceback (most recent call last):
File "/mnt/venv/cryptostore/lib/python3.7/site-packages/cryptofeed/feedhandler.py", line 230, in _connect
await self._handler(connection, handler)
File "/mnt/venv/cryptostore/lib/python3.7/site-packages/cryptofeed/feedhandler.py", line 259, in _handler
await handler(message, connection, self.last_msg[connection.uuid])
File "/mnt/venv/cryptostore/lib/python3.7/site-packages/cryptofeed/exchange/binance.py", line 321, in message_handler
await self._book(msg, pair, timestamp)
File "/mnt/venv/cryptostore/lib/python3.7/site-packages/cryptofeed/exchange/binance.py", line 231, in _book
await self._snapshot(exchange_pair)
File "/mnt/venv/cryptostore/lib/python3.7/site-packages/cryptofeed/exchange/binance.py", line 175, in _snapshot
response.raise_for_status()
File "/mnt/venv/cryptostore/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 1005, in raise_for_status
headers=self.headers,
aiohttp.client_exceptions.ClientResponseError: 429, message='Too Many Requests', url=URL('https://www.binance.com/api/v1/depth?symbol=CAKEBTC&limit=1000')
Apologies, I was running against cryptofeed-1.6.2. I have this setup on two different servers. cryptofeed-1.7.0 does resolve this issue.
yeah the issue is specific to book subscriptions because of the need to subscribe to the snapshots. Cryptofeed will need to be modified to reduce the number of requests issued at a time (something like a sleep between each rest request) or catch and handle the 429s (sleep when one is received). I dont subscribe to all symbols on any exchange, so its not a use case I personally encounter. It will eventually get fixed, but its not a high priority for me. Feel free to submit a PR, or I will eventually get to it.
@bmoscon how many pairs is it safe to use from your experience?
@bmoscon how many pairs is it safe to use from your experience?
Last time I checked the rate limit was something like 1200 requests per minute. However, I was never able to reach that amount without getting a 429 - most likely due to concurrently connecting to a large number of streams.