To reproduce the scraping, the following steps have to be taken:
- Get the urls to scrape:
get_urls.sh - Perform the scraping: see
scraping/README.md - Process the scraping results: see
processing/README.md(And possibly reschedule a scraping job) - Analyze the reults, see
analysis/README.md