lestrrat-p5 / HTML-RobotsMETA

Parse Robots META Directive

HTML::RobotsMETA parses HTML fails and look for robots directive inside
META tags

perldoc HTML::RobotsMETA for more info