A generic scraping engine that schedules, loads, runs crawlers and outputs to an external data store.
- Self contained pluggable scrapers
- Scraper execution engine that loads and executes the scrapers at pre-configured intervals
- Emit output to pre-configured data stores
- Web dashboard: admin, monitoring, and metrics
Pre-development