Environment variable for spiders needing proxy
jpmckinney opened this issue · comments
I hardcoded Montreal as using a proxy. Since the need for a proxy depends on the server, we can instead implement the list of spiders needing a proxy as an environment variable, and configure the custom settings on the base spider.
Would need to look into how/when Scrapy reads the custom_settings
class attribute, as we’ll want subclasses to be able to add to it.