This repo has fixed some spiders and other bugfixes related to compatibility with python3. The included spiders are confirmed to be working correctly in 2020.
This is a firmware scraper that aims to download firmware images and associated metadata from supported device vendor websites.
-
Configure the
firmware/settings.pyfile. Comment outSQL_SERVERif metadata about downloaded firmware should not be inserted into a SQL server. -
To run a specific scraper, e.g.
dlink:
scrapy crawl dlink
To run all scrapers with maximum 4 in parallel, using GNU Parallel:
parallel -j 4 scrapy crawl ::: `for i in ./firmware/spiders/*.py; do basename ${i%.*}; done`