trickstertwo / scrapy-cloudflare-middleware

A Scrapy middleware to bypass the CloudFlare's anti-bot protection

Scrapy "CloudFlare" middleware

A Scrapy middleware to bypass the CloudFlare's anti-bot protection, based on cloudflare-scrape.

Installation

pip install git+https://github.com/trickstertwo/scrapy-cloudflare-middleware.git

Usage

Add the middleware to your DOWNLOADER_MIDDLEWARES in settings.py

DOWNLOADER_MIDDLEWARES = {
    # The priority of 560 is important, because we want this middleware to kick in just before the scrapy built-in `RetryMiddleware`.
    'scrapy_cloudflare_middleware.middlewares.CloudFlareMiddleware': 560
}

Done. Happy scraping !