scrapy / scrapyd

A service daemon to run Scrapy spiders

https://scrapyd.readthedocs.io/en/stable/

scrapy/scrapyd Issues

SQLite queue is using all CPU on high frequency poller (<1s)
Closed a month ago12
Adding custom endpoints
Closed a month ago4
The work of scrapyd with the virtual environments of projects
Closed a month ago1
Add `--bind-address` and `--http-port` options (instead of configuration file)
Updated a month ago1
Enable configuration of bind_address, http_port, username and password via env vars
Closed a month ago1
When I am in Scrapyd deploy，always report an error
Closed 5 months ago1
Transparent list of projects
Closed 9 months ago7
Azure Feed exporter not working
Closed 8 months ago2
Bad installation steps for "python setup.py install"
Closed 9 months ago2
Add support for environment variables
Closed 9 months ago7
Error on windows when trying to deploy - [WinError 32]
Closed 9 months ago17
May I ask is there any way use scrpyd to run python script with CrawlerProcess
Closed a year ago2
Cannot access environment variables in the spider running under ScrapyD
Closed a year ago2
Run tests on Windows & better Windows support
Updated a year ago
How to pause and resume crawling with scrapyd?
Closed a year ago3
Use wheel files for deployment
Closed a year ago1
xxx.log can't be found
Closed a year ago3
Add reverse proxy support for web interface
Closed a year ago1
[Feature Request] S3 Storage
Closed a year ago3
Delete job from finished jobs
Updated a year ago12
ReadTheDocs build
Closed a year ago2
crawl: error: Invalid -s value, use -s NAME=VALUE
Closed a year ago9
`priority` is rejected as a spider argument
Closed a year ago5
scrapyd writing logs to console instead of log files in docker container
Closed a year ago9
Replace `FEED_URI` and `FEED_FORMAT` with `FEEDS` in feedexporter
Closed a year ago1
encoding errors when running a spider via scrapyd
Closed a year ago2
API - add items and logs url to listjobs.json
Closed a year ago3
Replace distutils.version.LooseVersion with packaging.version Version
Closed a year ago1
Print the release version when running with the "--version" flag.
Closed a year ago2
High frequency scheduling
Closed a year ago1
Cancel button redirects to another page - and this makes the user experience a bit tiresome
Updated a year ago
facing unexpected builtins.TypeError error when running >Scrapyd command .
Closed a year ago6
My scrapyd output files sudden change to /root
Closed a year ago2
jobs not being removed from running status
Closed a year ago3
How to correctly configure CONCURRENT_REQUESTS in a project with multiple spiders
Closed a year ago1
add FEEDS support like "curl http://localhost:6800/schedule.json ... -d setting=FEEDS=output.csv"
Closed a year ago
Why not use __main__?
Closed 2 years ago3
Persistent scheduling based on CRON
Closed 2 years ago4
upload egg file raise an DistributionNotFound exception for requirements package
Closed 2 years ago1
Use JOBDIR of scrapy
Closed 2 years ago2
should w3lib be declared in configuration files?
Closed 2 years ago
readthedocs docs not updated
Closed 2 years ago1
Hooks to call some url when scraping completes
Closed 3 years ago1
Scrapyd creating new egg files in .cache every time a job is scheduled.
Updated 3 years ago4
Cross origin problem
Closed 3 years ago2
Can't create custom webservice through scrapyd
Closed 3 years ago1
apt-get install scrapyd doesn't work on Ubuntu #142
Closed 3 years ago3
how to start two case on a PC with diffirent port
Closed 3 years ago1
I'm experiencing the same issue, scrapyd is accessible from within the container, but not from the outside.
Closed 3 years ago4
scrapyd schedule API with environment variables
Closed 3 years ago1