birdspider
web scrawler for a birds website.
Description
This project is base on Scrapy.
Install
pip install Scrapy
, virtualenv
is highly recommended. If you encounter any problem, please refer to the official document of Scrapy Installation.
Usage
cd
to theroot
directory- run
scrapy runspider myspider.py
in shell - find the
birds.json
file in the same directory, all scrawled data are in it - enjoy :)
Modification
Change target website and target scrawl pages by change this line.