AntoineGagne / robots

A parser for robots.txt with support for wildcards. See also RFC 9309.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Add a function to return which agent

AntoineGagne opened this issue · comments

With an opaque type, it could be used when looking up if an URL is allowed/disallowed instead of having to look for the agent in the rules everytime.