web-robots

There are 0 repository under web-robots topic.

• jonasjacek / robots.txt

  Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.

  googlebotbingbotrobots-txtrobots-exclusion-standardblocking-botsuser-agentweb-robotsseosearch-enginewhitelistcrawlersweb-crawlingcrawlingsearch-engine-optimizationbaiduspidertwitterbot
  84

• din0s / ml-for-bot-detection

  A Python notebook showcasing the use of Machine Learning for the task of bot detection, with an emphasis on e-commerce sites.

  machine-learningweb-robotsbot-detectione-commerce
  Language:Jupyter Notebook 13

• acuciureanu / spidertrap-rs

  A simple trap for web crawlers

  anti-botbot-detectioncybersecuritydata-privacyethical-hackingintrusion-detectionnetwork-securityspider-trapweb-crawlerweb-monitoringweb-robotsweb-scrapingweb-securityweb-spiderwebsite-protectionanti-scrapingbot-trapcrawler-trapserver-defenserust
  Language:Rust 12

• jimsmart / progszy

  Progszy is a hard-caching HTTP(S) proxy server, for web robots.

  https-proxyhttp-proxycaching-proxyweb-robotsgo
  Language:Go 1