scrapinghub / scmongo

MongoDB extensions for Scrapy

scmongo

This project contains some extensions for using MongoDB with the Scrapy web crawling framework.

Requirements

• Scrapy 0.14 or above
• pymongo 2.4 or above

Install

Download and run: python setup.py install

Available extensions

Mongo Cache Storage

Module: scmongo.httpcache

A MongoDB backend for HTTP cache storage. It stores responses using GridFS.

To use it, set the following Scrapy setting in your project:

HTTPCACHE_STORAGE = 'scmongo.httpcache.MongoCacheStorage'