fooock / robots.txt

:robot: robots.txt as a service. Crawls robots.txt files, downloads and parses them to check rules through an API

https://robotstxt.io

fooock/robots.txt Issues

Comparison method violates its general contract!
Closed 4 years ago
Handle redirects
Closed 5 years ago
Save sitemaps and crawl delay
Closed 5 years ago
Import Postman collection
Closed 5 years ago
Add header to identify robots.txt crawler
Closed 5 years ago
Limit number of urls to update at the same time
Closed 5 years ago
Expose method to find rules by host in crawler
Closed 5 years ago
If we check an empty path, the result is always true
Closed 5 years ago
Use request user-agent if no one was provided
Closed 5 years ago
Handle NumberFormatException in parser
Closed 5 years ago
Use HTTP headers to control host rule expiration
Updated 5 years ago1
URL matching based on wildcards for path values
Closed 5 years ago1
Change docker base image
Closed 5 years ago
Limit container resources
Updated 5 years ago
Auth based in JWT
Updated 5 years ago
Send url to crawl if not exists when check
Closed 5 years ago
Rate limit for the checker API
Updated 5 years ago
Check if there is more than one group with same user agent
Closed 5 years ago1
Create redis consumer groups
Closed 5 years ago1
Change standalone redis server
Closed 5 years ago
Put urls received from kafka to a redis queue
Closed 5 years ago1
Cleanup constructors from code
Closed 5 years ago
Strategy to crawl robots from ftp scheme
Updated 5 years ago
Error getting stack overflow robots.txt file
Closed 5 years ago
Limit robots.txt maximum file size
Closed 5 years ago
Ignore content types not equals to text/plain
Closed 5 years ago
Save followed URL when a redirection occurs
Closed 5 years ago
Check disallow typos?
Updated 5 years ago
Update spring boot to 2.1.6
Closed 5 years ago
Implement robots.txt download priority system
Updated 5 years ago
Update gradle wrapper to 5.4.0
Closed 5 years ago