robots-tag

There are 1 repository under robots-tag topic.

• VIPnytt / UserAgentParser

  User-Agent parser for robots.txt, X-Robots-tag and Robots-meta-tag rule sets

  crawlerreprobotrobots-exclusion-protocolrobots-meta-tagrobots-tagrobots-txtspideruser-agentuser-agent-parserx-robots-tag
  Language:PHP 1

• RobotstxtGenerator / Robots.txt-Generator

  The Robots.txt Generator tool helps you to create the Robots.txt file for your website.

  robotsrobots-txtrobots-generatorrobots-tagrobots-txt-generatorrobots-txt-creatorrobots-txt-file
  0

• toimik / RobotsProtocol

  Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag

  robots-txtrobots-parserrobotstxtrobotsrobots-exclusion-protocolrobots-exclusion-standardrobots-tagx-robots-tagmeta-tags
  Language:C# 0