HTML::RobotsMETA parses HTML fails and look for robots directive inside META tags perldoc HTML::RobotsMETA for more info
Parse Robots META Directive
HTML::RobotsMETA parses HTML fails and look for robots directive inside META tags perldoc HTML::RobotsMETA for more info
Parse Robots META Directive