robots.txt parser for Rust.
Add it to your Cargo.toml
:
[dependencies]
robotparser = "0.10"
Add extern crate robotparser
to your crate root and your're good to go!
extern crate robotparser;
use robotparser::RobotFileParser;
fn main() {
let parser = RobotFileParser::new("http://www.python.org/robots.txt");
parser.read();
assert!(parser.can_fetch("*", "http://www.python.org/robots.txt"));
}
This work is released under the MIT license. A copy of the license is provided in the LICENSE file.