Giters
anjackson
/
scrapy-url-frontier
A Scrapy module for URL Frontier integration
Geek Repo:
Geek Repo
Github PK Tool:
Github PK Tool
Stargazers:
1
Watchers:
2
Issues:
5
Forks:
anjackson/scrapy-url-frontier Issues
Reduce logging, report stats instead.
Updated
2 years ago
Add 'source' metadata field that is the submitted URL
Updated
2 years ago
Create initial URL Frontier scheduler with distributed crawl support
Closed
2 years ago
Re-route URLs when partitions change
Updated
2 years ago
Add support to use URL Frontier as a Frontera back-end
Updated
2 years ago