idrodriguez / scrapyd-auth

Scrapy daemon running in an authenticated way through nginx

Scrapyd with nginx authentication

Scrapy daemon is not developed as a daemon that allows authentication for access. Therefore, to protect your daemon, you need to serve scrapyd through a web server like nginx. Bypassing the request to scrapyd internally and returning the output to nginx.

In this setup, nginx is working on port 80 and scrapyd in port 6800; externally, you will need to map only the port 80 as is mentioned in the docker-compose.yml file.

logparser is needed if you are using ScrapydWeb for managing your projects. It is included and running in the docker instance, communicating directly with scrapyd.

.env file

You must setup your .env file accordingly to what your spider is requesting. For authentication purposes only, use:

USERNAME=USERNAME
PASSWORD=PASSWORD

After the setup you can reference in the docker-compose.yml file.

DockerHub Images

You can find the image scrapyd-auth in dockerhub.

Feedback

Issues

If you have any problems with or questions about this image, please file an issue on the GitHub repository.

Contributing

Pull requests welcome :-)