Disallow: // shouldn't behave the same as Disallow: /
wumpus opened this issue · comments
https://www.ibm.com/robots.txt has DIsallow:// and it appears that they expect it to not disallow everything. reppy treats it the same as Disallow:/
Modern robots.txt Parser for Python
wumpus opened this issue · comments
https://www.ibm.com/robots.txt has DIsallow:// and it appears that they expect it to not disallow everything. reppy treats it the same as Disallow:/