scrapymongo
Several mongo component for scrapy framework.
Features
* Mongo Pipeline with Foreign key support
* Monog HttpCache Storage
Design
MongoPipeline
- each item type is mapped to a collection. collection name come from class of Item.
- each item is a record of the collection above. with new generated id. and a 'key' property, whose value is unique for every scrapped item. the item class have to ensure the value of key exists.
- foregin key support. in Item defind fk field, pipeline will calculate the fk_id ('type'+'_id') Field({ 'fk' : True }) Field({'fk': {'type':'class_name','key':'key_in_type_in_the_left'}})
MongoCacheStorage
Settings options
MONGO_PIPELINE_HOST - mongo host, defualt is 'localhost'
MONGO_PIPELINE_KEYS - 'key' property of item. checked in order. default is ['key', 'url']. i.e. if 'key' doesn't exist , check 'url'
MONGO_PIPELINE_DBNAME - mongo db name, default is None. if None, pipeline will use scrapy project name as db name
MONGO_PIPELINE_DBNAME_BOTSUFFIX - if set True, use DBNAME.BOTNAME as mongo database name. only take effect when if DBNAME is exlicitly set.
MONGO_PIPELINE_COLNAME_BOTPREFIX - if set True, use BOTNAME.ITEM-CLASS-NAME as mongo collection name.x
License
MIT