pbs ignores server rate limit
sgerke opened this issue · comments
the pbs tool does not consider rate limits set by the server but rather assumes the default rate limit of 300. This makes e.g. creating more than 300 tasks at once very time consuming.
I see this issue is starving. I can contribute but I would like to know if 1) it is desired and 2) what approach to follow.
From the code and this page https://docs.pybossa.com/api/intro/#rate-limiting I believe that a reasonable approach would be to ask the server for the total limit and use it to calculate the sleep interval. The drawback is that you have to open a new connection.
In any case, I would put a configuration parameter for the server, so the user could overwrite the value.
Yes, this needs to be fixed. The solution will be to use a pre-flight request to get the X-Rate-Limit header. Then based on that, auto-adjust it.
The API returns the deadline about when to renew the window and remaining requests that you can do to a specific endpoint. @corrado9999 please send any PR with this approach and we will review it and merge it.