McTwist / Blockland-Forum-Add-On-Scraper

Scrapes for Add-Ons on Blockland forum

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Optimize queue

McTwist opened this issue · comments

Currently the work queue is used in the way data is inserted. This results in long waiting if trying to connect to same domain, as the domain blocker will wait.

Instead, add a priority queue(heap) which will, instead of waiting, put the item back in the queue and sort itself depending on the priority. Keep in mind that this priority is changing each iterarion as ut needs to know what already exists within it to determine when to be called.

Sadly, futures module does not have this kind of feature, so we have to create our own threadpool to be able to pull this off.