There are 0 repository under web-robots topic.
Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.
A simple trap for web crawlers
A Python notebook showcasing the use of Machine Learning for the task of bot detection, with an emphasis on e-commerce sites.