scrapy-fake-useragent
Random User-Agent middleware based on
fake-useragent. It
picks up User-Agent
strings based on usage
statistics
from a real world database.
Configuration
Turn off the built-in UserAgentMiddleware
and add
RandomUserAgentMiddleware
.
In Scrapy >=1.0:
DOWNLOADER_MIDDLEWARES = { 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None, 'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400, }
In Scrapy <1.0:
DOWNLOADER_MIDDLEWARES = { 'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None, 'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400, }