Optimize queue
McTwist opened this issue · comments
Currently the work queue is used in the way data is inserted. This results in long waiting if trying to connect to same domain, as the domain blocker will wait.
Instead, add a priority queue(heap) which will, instead of waiting, put the item back in the queue and sort itself depending on the priority. Keep in mind that this priority is changing each iterarion as ut needs to know what already exists within it to determine when to be called.
Sadly, futures module does not have this kind of feature, so we have to create our own threadpool to be able to pull this off.