dfdeshom / scrapy-kafka

Kafka-based components for Scrapy

scrapy-kafka

Kafka-based components for Scrapy. There are 2 components:

• A custom Spider that waits for URLs to crawl via a Kafka topic. When there are no more messages to read for the topic, the Spider just stays idle.
• A custom ItemPipeline component that stores a JSON-ified Item back into another Kafka topic.

Please see the example directory for how to use this.

Contributors

Contributors to scrapy-kafka, listed alphabetically:

• Matthew Daniel @mdaniel
• Didier Deshommes @dfdeshom