robots-exclusion-standard

There are 1 repository under robots-exclusion-standard topic.

• samclarke / robots-parser

  NodeJS robots.txt parser with support for wildcard (*) matching.

  user-agentjavascriptnodejsrobots-txtrobots-exclusion-standardrobots-parser
  Language:JavaScript 141

• jonasjacek / robots.txt

  Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.

  googlebotbingbotrobots-txtrobots-exclusion-standardblocking-botsuser-agentweb-robotsseosearch-enginewhitelistcrawlersweb-crawlingcrawlingsearch-engine-optimizationbaiduspidertwitterbot
  83

• samclarke / robotstxt

  Go robots.txt parser

  robots-txtrobotstxtrobots-parserrobots-exclusion-standardgolang
  Language:Go 16

• andreburgaud / robotspy

  Alternative robots parser module for Python

  pythonrobotstxtrobots-parserrobots-exclusion-protocolrobots-exclusion-standardparser
  Language:Python 14

• chrisakroyd / robots-txt-parser

  A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

  robots-parserrobots-exclusion-standardrobots-txtnode-jsjavascriptparserrobots-txt-parser
  Language:JavaScript 11

• crwlrsoft / robots-txt

  Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

  robots-txt-parserweb-crawlinghacktoberfestrobots-txtrobots-exclusion-standardrobots-exclusion-protocolweb-scraping
  Language:PHP 8

• hgruniaux / robotstxt

  The repository contains Google-based robots.txt parser and matcher as a C++ library (compliant to C++17).

  robots-txtrobots-parserrobots-exclusion-protocolrobotstxtrobots-exclusion-standard
  Language:C++ 0

• toimik / RobotsProtocol

  Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag

  robots-txtrobots-parserrobotstxtrobotsrobots-exclusion-protocolrobots-exclusion-standardrobots-tagx-robots-tagmeta-tags
  Language:C# 0

• 0xIbra / robots-txt-component

  Fully native robots.txt parsing component without any dependencies.

  robots-txtrobots-parserrobots-exclusion-standardnodejsrobots-noderobots-txt-node
  Language:JavaScript