Robot name comparison must be case-insensitive
bortzmeyer opened this issue · comments
Currently, the match of the robot name (to User-Agent) is done in a case-sensitive way. RFC 9309, section 2.2.1 says it must be case-insensitive.
AntoineGagne / robots
A parser for robots.txt with support for wildcards. See also RFC 9309.