jonasjacek / robots.txt

Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.

https://www.ditig.com/publications/robots-txt-template

jonasjacek/robots.txt Stargazers