Giters
chriskite
/
robotex
Ruby library to obey robots.txt
Geek Repo:
Geek Repo
Github PK Tool:
Github PK Tool
Stargazers:
20
Watchers:
4
Issues:
6
Forks:
8
chriskite/robotex Issues
Warnings about unused variable and instance variable not initialized
Closed
5 years ago
Maintenance status?
Updated
6 years ago
Comments count
3
Robotex does not follow user-agent precedence rules
Updated
6 years ago
Robotex does not follow redirects
Updated
6 years ago
method to extract sitemaps
Updated
6 years ago
Robotex does not support comments
Updated
7 years ago