crwlrsoft / robots-txt

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

Robots Exclusion Standard/Protocol Parser

for Web Crawling/Scraping

Use this library within crawler/scraper programs to parse robots.txt files and check if your crawler user-agent is allowed to load certain paths.

Documentation

You can find the documentation at crwlr.software.

Contributing

If you consider contributing something to this package, read the contribution guide (CONTRIBUTING.md).