scrapy / scrapyd

A service daemon to run Scrapy spiders

Home Page:https://scrapyd.readthedocs.io/en/stable/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

High frequency scheduling

pspsdev opened this issue · comments

In a scenario where new job is added multiple times every second on a strong server what is the best polling rate? It seems if the polling rate is too high it will overload the cpu just because of the polling rate, if the polling rate is too low then the jobs are not starting as fast and server is wasting resources.

Maybe setting a queue to be rabbitmq would fix the high cpu usage in high frequency polling?