TurnerSoftware / RobotsExclusionTools

A "robots.txt" parsing and querying library for .NET

TurnerSoftware/RobotsExclusionTools Issues

New Robots meta tag rules for snippets
Closed 2 years ago1
Benchmarking (Paths and Wildcards)
Closed 2 years ago3
Disallow: not working without a space
Closed 2 years ago
Enhancing Tokenization and Parsing
Closed 2 years ago1
Error processing one of robots.txt
Closed 4 years ago1
The ExclusionTools throws a WebException while requesting Robots.txt-File
Closed 4 years ago11
Add API for parsing X-Robots-Tag
Closed 5 years ago2
Handling failed validation during parsing
Closed 5 years ago1
Benchmarking (String vs Stream)
Closed 5 years ago6