Sleep mechanism for github ban
luyaor opened this issue · comments
When I was using github-flask to crawl all the fork list of one repo on multi-thread:
request('GET', 'repos/%s/forks' % project_name, True)
It wil be ban by github.
Is there any way I can avoid this if I use all_pages=True in
request(method, resource, all_pages=False, **kwargs) ?
My idea is that give user a option for not crawling so fast, like sleep(1) after each API call.
Currently, there is no mechanism in the library for that. A PR would be welcome.
You can check X-Ratelimit headers in response before making a new call. See https://developer.github.com/v3/#rate-limiting