binux / pyspider

A Powerful Spider(Web Crawler) System in Python.

Home Page:http://docs.pyspider.org/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

can us make scheduler simplex,let processor handle taskdb?

JiangYinyuan12 opened this issue · comments

hi:
i should pending 20w urls every hour,but only scheduler which is limited one process can handle taskdb,so it is realy high load for scheduler.
but i think processor can also handle taskdb using its multi process advantage,i have change the code,and running online for more than 4 month.
i think let scheduler's duty more and more less is very important,because it's the key bottleneck of the all system.
can community use this thought?