austinsonger / sitemapsandrobotsaroundtheweb

Sitemaps and Robots.txt for websites around the world.

Sitemaps and Robots around the Web

Robots.txt for websites around the world.

This is to be used for educational purposes and Security Researchers and Bug Bounty Programs.

Robots.txt

US Companies

• Amazon
• Apple
• Facebook
• Github
• Gitlab
• Google
• Target
• Walmart

Sitemaps

Only add sitemaps that are reasonable in size.

• Google

How To

• Robots.txt Basics
• Sitemap Basics

Contributing

Any contributions you make are greatly appreciated.

1. Fork the Project
2. Commit your Changes
3. Push to the Branch
4. Open a Pull Request