Spread multiple requests to a single domain evenly inside an update batch
brutasse opened this issue · comments
Currently updates are scheduled naively: take 1/12th of the queue, schedule, run N workers in parallel to consume the queue.
If there are lots of reddit.com urls in a batch, some may get rate-limited very quickly.
Need to find a way to schedule a batch sequentially with proper rate-limit control instead of scheduling all urls at the same time.
Worked around in 3434ad7