Update Documentation
get-data- opened this issue · comments
Good Day,
It seems like the method for determining the robots.txt has changed and the documentation needs updating. Please see below.
Determining the robots.txt URL
old
Robots.robotsUrl('http://userinfo@example.com:8080/path;params?query#fragment')
new
Robots.robots_url('http://userinfo@example.com:8080/path;params?query#fragment')