Version 2.0 map

[Jan-Petter Gundersen]

I think it's time to feature freeze v.1.0. Every robots.txt standard is supported, and it might just need some code cleanup before it is ready for release.

Just want to share some dev notes for what to expect in version 2.0. The development haven't started yet, but soon it will. Here is the list:

• Request the HTTP headers of the provided URL, detecting any redirects
• Download the correct robots.txt file automatically
  • Option 1: cURL
  • Option 2: Guzzle https://github.com/guzzle/guzzle
  • Option 3: ???
• X-Robots-Tag HTTP header specification support
• Improve the encoding detection
• Simplified usage. Fewer lines, more checks and more features.
• Provide a tool for robots meta tag specification support
• Caching system for robots.txt files - Scheduled for version 2.1

Feel free to make suggestions or discuss these changes :-)