Forked Version of the Google Robots.txt Parser and Matcher Library
This is a lightly modified version of the library, just changing the output of the included binary to a JSON like string.
Like the main library, some further implementation would be needed to account for implementation, for example if attempting to mimic Google's behaviour, you would need to account for a user-agent of Googlebot-Image
falling back to using Googlebot
if no specific matching agent is found, or if you wanted to mimic applebot
, it too falls back to googlebot
if a specific User-agent is not found.
Google Robots.txt Parser and Matcher Library
The repository contains Google's robots.txt parser and matcher as a C++ library (compliant to C++11).
About the library
The Robots Exclusion Protocol (REP) is a standard that enables website owners to control which URLs may be accessed by automated clients (i.e. crawlers) through a simple text file with a specific syntax. It's one of the basic building blocks of the internet as we know it and what allows search engines to operate.
Because the REP was only a de-facto standard for the past 25 years, different implementers implement parsing of robots.txt slightly differently, leading to confusion. This project aims to fix that by releasing the parser that Google uses.
The library is slightly modified (i.e. some internal headers and equivalent symbols) production code used by Googlebot, Google's crawler, to determine which URLs it may access based on rules provided by webmasters in robots.txt files. The library is released open-source to help developers build tools that better reflect Google's robots.txt parsing and matching.
For webmasters, we included a small binary in the project that allows testing a single URL and user-agent against a robots.txt.
Building the library
Quickstart
We included with the library a small binary to test a local robots.txt against a user-agent and URL. Running the included binary requires:
- A compatible platform (e.g. Windows, macOS, Linux, etc.). Most platforms are fully supported.
- A compatible C++ compiler supporting at least C++11. Most major compilers are supported.
- Git for interacting with the source code repository. To install Git, consult the Set Up Git guide on GitHub.
- Although you are free to use your own build system, most of the documentation within this guide will assume you are using Bazel. To download and install Bazel (and any of its dependencies), consult the Bazel Installation Guide
Building with Bazel
Bazel is the official build system for the library, which is supported on most major platforms (Linux, Windows, MacOS, for example) and compilers.
To build and run the binary:
$ git clone https://github.com/google/robotstxt.git robotstxt
Cloning into 'robotstxt'...
...
$ cd robotstxt/
bazel-robots$ bazel test :robots_test
...
/:robots_test PASSED in 0.1s
Executed 1 out of 1 test: 1 test passes.
...
bazel-robots$ bazel build :robots_main
...
Target //:robots_main up-to-date:
bazel-bin/robots_main
...
bazel-robots$ bazel run robots_main -- ~/local/path/to/robots.txt YourBot https://example.com/url
user-agent 'YourBot' with url 'https://example.com/url' allowed: YES
Building with CMake
CMake is the community-supported build system for the library.
To build the library using CMake, just follow the steps below:
$ git clone https://github.com/google/robotstxt.git robotstxt
Cloning into 'robotstxt'...
...
$ cd robotstxt/
...
$ mkdir c-build && cd c-build
...
$ cmake .. -DROBOTS_BUILD_TESTS=ON
...
$ make
...
$ make test
Running tests...
Test project robotstxt/c-build
Start 1: robots-test
1/1 Test #1: robots-test ...................... Passed 0.02 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 0.02 sec
...
$ robots ~/local/path/to/robots.txt YourBot https://example.com/url
{
"allowed": true,
"line": 3,
"agent_specific": false
}
allowed
- true OR false - If the URL is allowed for crawlingline
- Which line of the robots.txt applies. Returns 0 if no rule matchedagent_specific
- If the rule matched was agent specific, i.e. a testrobots ~/local/path/to/robots.txt Foo https://example.com/url
foundUser-agent: Foo
in the robots.txt file (true
), or it fell back toUser-agent: *
(false
)
Notes
Parsing of robots.txt files themselves is done exactly as in the production version of Googlebot, including how percent codes and unicode characters in patterns are handled. The user must ensure however that the URI passed to the AllowedByRobots and OneAgentAllowedByRobots functions, or to the URI parameter of the robots tool, follows the format specified by RFC3986, since this library will not perform full normalization of those URI parameters. Only if the URI is in this format, the matching will be done according to the REP specification.
License
The robots.txt parser and matcher C++ library is licensed under the terms of the Apache license. See LICENSE for more information.
Links
To learn more about this project:
- check out the internet draft,
- how Google's handling robots.txt,
- or for a high level overview, the robots.txt page on Wikipedia.