Use this Scrapy to obtain bill data from the NC Legislature website.
The scrapy extracts each bill's data into an object. Use scrapy command to out put a JSON list of bill objects.
- Python 3
- Environment variables stored for:
- AWS Access Key (
S3_ACCESS_KEY
) - AWS Secret Key (
S3_SECRET_KEY
) - AWS S3 Bucket (
S3_BUCKET_NAME
)
- psycopg2 intalled
pip install psycopg2
- tinys3 installed
pip install tinys3
- Requires python3
- Install Scrapy, pip for example:
pip install scrapy
- Navigate into repo
- Tell Scrapy to crawl "bills"
scrapy crawl bills -o <filename>.<ext>
(. is a JSON file that will be created at runtime and will contain the extracted data.)