A sane, minimal robots.txt file (for the western world) You're unlikely to need any other crawlers on your site to be found. If you have special requirements or need/want to use local search engines, this is probably more of a starting point for you. As this blocks all but the excepted user agents, it should plainly identify crawlers that do not respect the settings.