RobotReader
Robot.txt reader (suitable for spider/crawler)
Written by Seyed Vahid Hashemi
This java class can be using inside your spider to read robot.txt throughly
feel free to complete it more!
USAGE
RobotReader r = new RobotReader("YOUR SPIDER SIGNATURE");
readRobot(new URL("http://www.somewebsite.com"));
boolean answer = isUrlAllowed("/url/");