open-contracting / kingfisher-collect

Downloads OCDS data and stores it on disk

Home Page:https://kingfisher-collect.readthedocs.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Environment variable for spiders needing proxy

jpmckinney opened this issue · comments

I hardcoded Montreal as using a proxy. Since the need for a proxy depends on the server, we can instead implement the list of spiders needing a proxy as an environment variable, and configure the custom settings on the base spider.

Would need to look into how/when Scrapy reads the custom_settings class attribute, as we’ll want subclasses to be able to add to it.