Add --ignore-robots option
brendt opened this issue · comments
With the latest version of the crawler, it will take robots rules into account. There should be an option to ignore them.
Implemented and released.
spatie / http-status-check
CLI tool to crawl a website and check HTTP status codes