#scrapy scrapy projects
use scrapy list to see the spiders
to run scrapy from python shell but not scrapy commond, use python run.py
queueManager.py --> global Queue manager server
urlReader.py --> python script to read urls from given path into global queue
run.py --> scrapy spiders to crawl urls from global queue
Usage:
1.run python queueManager.py as a background service
2.run python urlReader.py to read urls
3.run python run.py to crawl urls