robotstxt
There are 0 repository under robotstxt topic.
- politeLanguage:R 324
alextim / astro-lib
Makes it easy to add robots.txt, sitemap and web app manifest during build to your Astro app.
Language:TypeScript 99- Language:Rust 85
- Language:R 69
itgalaxy / generate-robotstxt
Generator robots.txt for node js
Language:JavaScript 64A webpack plugin to generate a robots.txt file
Language:JavaScript 35elchiconube / generate-robots-txt
The right robots.txt file for your project
Language:TypeScript 21- Language:Go 16
andreburgaud / robotspy
Alternative robots parser module for Python
Language:Python 15herrbischoff / robots.txt
A sane, minimal robots.txt file (for the western world)
jasonmajors / robotstxt
This is a small package to make parsing robots.txt rules easier.
Language:PHP 3- 2022-ccd-sips
spcanelon / 2022-ccd-sips
Workshop materials for creating the Center City Sips District 2022 interactive map
Language:HTML 3 aafeher / go-sitemap-parser
Go language library for parsing Sitemaps
Language:Go 2- Language:HTML 2
- Language:Rust 2
vxern / robots_txt
⚙️ A quality `robots.txt` ruleset parser to ensure your application follows the standard specification for the file.
Language:Dart 2ChrisWinters / robotstxt-manager
A Simple Robots.txt Manager Plugin For Wordpress.
Language:PHP 1littlebizzy / virtual-robotstxt
Replaces the default virtual robots.txt generated by WordPress with an editable one, and deletes any physical robots.txt file that may already exist.
- Language:Elm 1
MaximeGuinard / Robots.txt-Viewer
🌐 Displays the contents of robots.txt and sitemap.xml files of a website google extension
Language:JavaScript 1nicholasbergesen / RobotsParser
Parse robots.txt and traverse sitemaps.
Language:C# 1- Language:Python 1
- Language:TypeScript 1
TahaT80 / Robots_Scanner
Robots Scanner
Language:Python 1Zer0Divis0r / rebuild-cache-sitemap
GitHub action to rebuild CDN cache according to sitemaps
Language:Shell 1ananthram-g / WebScraping
A Web Scraping project using robotstxt & rvest in R programming.
Language:R 0hgruniaux / robotstxt
The repository contains Google-based robots.txt parser and matcher as a C++ library (compliant to C++17).
Language:C++ 0KnuckleheadsClub / rbdt
rbdt is a python library (written in rust) for parsing robots.txt files for large scale batch processing.
Language:Rust 0ossama131 / Bias-to-Search-Engines-from-Robots.txt
Determining bias to search engines from Robots.txt
Language:Jupyter Notebook 0toimik / RobotsProtocol
Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag
Language:C# 0ajayjsr / bloggerrobotstxtgenerator
The Blogger Robots.txt Generatois a tool designed to simplify the process of creating a robots.txt file for websites hosted on the Blogger platform. A robots.txt file is crucial for controlling how search engines index your site. This generator allows users to customize and generate a robots.txt file tailored to their specific needs.
Language:HTMLdemyanovs / robotstxt
RobotsTXT is a robots.txt parser written in Go. The parser reads and interprets robots.txt files into a struct.
Language:Go- Language:Go
peaceiris / docker-images
A collection of Docker images: robotstxt, linuxbrew, gcloud, and psql
Language:Makefileprod3v3loper / php-honeypot-robots-bait
🍯 HoneyPot Robots Bait
Language:PHP
