RobotScraper is a simple tool written in Python to check each of the paths found in the robots.txt file and what HTTP response code they return.
Geek Repo:Geek Repo
Github PK Tool:Github PK Tool