justinvw / officiele-bekendmakingen-scraper

Scrapes the search result pages of https://zoek.officielebekendmakingen.nl/ using Scrapy, and downloads the XML documents it may find along it's way.

Officiële bekendmakingen scraper

Author: Justin van Wees ([e-mail])
Date: 2010-06-21

Officiële bekendmakingen scraper scrapes the search result pages of https://zoek.officielebekendmakingen.nl/ and downloads the XML documents it may find along it's way.

Requirements

• Python 2.5 or 2.6
• Twisted 2.5.0, 8.0 or above
• libxml2 2.6.28 or above
• Scrapy 0.8

Installation and configuration

After you've made sure that all the required Python packages are installed, please edit "officielebekendmakingen/settings.py". The settings should be self explanatory.

Running Officielebekendmakingen scraper

Run python scrapy-ctl.py crawl zoek.officielebekendmakingen.nl

You can monitor the Scrapy process by visiting http://[HOSTNAME]:6080 or by opening a Telnet session to port 6023 (the "stats" object contains information about the current run)