lazopm / webscrap-sample

scraping millions of records from a public government website

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Web Scraping Sample

A script and tools to scrape a whole country's population data from a web service and put them in a database.
Threaded to make the ~19 million records faster to download.
Separate script to verify downloaded records and a small Flask app to view the data.

About

scraping millions of records from a public government website


Languages

Language:Python 85.1%Language:HTML 14.9%