Java API and parser for robots.txt, as described in http://www.robotstxt.org/wc/norobots-rfc.html.
Fork from project OSJava Norbert project https://code.google.com/p/osjava/downloads/detail?name=norbert-0.3.2-src.zip&can=2&q=
Java robots.txt file parser an API
Java API and parser for robots.txt, as described in http://www.robotstxt.org/wc/norobots-rfc.html.
Fork from project OSJava Norbert project https://code.google.com/p/osjava/downloads/detail?name=norbert-0.3.2-src.zip&can=2&q=
Java robots.txt file parser an API
BSD 3-Clause "New" or "Revised" License