robots-parser
There are 2 repositories under robots-parser topic.
samclarke / robots-parser
NodeJS robots.txt parser with support for wildcard (*) matching.
Language:JavaScript 141- Language:DIGITAL Command Language 51
VIPnytt / RobotsTxtParser
An extensible robots.txt parser and client library, with full support for every directive and specification.
Language:PHP 24- Language:Rust 17
- Language:Go 16
fooock / robots.txt
:robot: robots.txt as a service. Crawls robots.txt files, downloads and parses them to check rules through an API
Language:Java 15andreburgaud / robotspy
Alternative robots parser module for Python
Language:Python 14- Language:Elixir 13
chrisakroyd / robots-txt-parser
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
Language:JavaScript 11b4dnewz / robots-parse
A lightweight and simple robots.txt parser in node
Language:TypeScript 7eliasdabbas / robotstxt_app
Visual App for Testing URLs and User-agents blocked by robots.txt Files
Language:Python 4larevanchedessites / google-robotstxt-ruby
🤖 Ruby gem wrapper around Google Robotstxt Parser C++ library
Language:Ruby 4AntoineGagne / robots
A parser for robots.txt with support for wildcards. See also RFC 9309.
Language:Erlang 3- Language:Go 3
mounicmadiraju / robot.txt-changes
💧 Test your robots.txt with this testing tool. Check if a URL is blocked, which statement is blocking it and for which user agent. You can also check if the resources for the page (CSS and JavaScript) are disallowed!. Robots.txt files help you guide how search engines crawl your site, and can be an integral part of your SEO strategy.
Language:Python 2muratgozel / robotstxt-util
RFC 5234 spec compliant robots.txt builder and parser. 🦾
Language:TypeScript 2nicholasbergesen / RobotsParser
Parse robots.txt and traverse sitemaps.
Language:C# 1- Language:TypeScript 1
- Language:C# 0
hgruniaux / robotstxt
The repository contains Google-based robots.txt parser and matcher as a C++ library (compliant to C++17).
Language:C++ 0menno120 / robots-parser
Typescript robots.txt parser with support for wildcard (*) matching.
Language:JavaScript 0monthly-coding / robotstxt-python
robots.txt validator for python
0toimik / RobotsProtocol
Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag
Language:C# 00xIbra / robots-txt-component
Fully native robots.txt parsing component without any dependencies.
Language:JavaScriptptsochantaris / can-proceed
A small, tested, no-frills parser of robots.txt files in Swift.
Language:Swiftthomasleveil / pyrobots
python binding for Google robots.txt parser C++ library
Language:Python
