Option for disable redirects

[Sezgin]

Hello,

It seems that reppy follows redirects and if redirect's target is a 404 page, it tries to fetch same robots.txt over and over again while using RobotsCache. It may be very useful if you can add "do not follow redirects" option.

Thank you very much :)

[Sezgin]

I found that it's already possible by passing requests options to RobotsCache

Example usage for who may need it in future;

robots_cache = RobotsCache(timeout=1, allow_redirects=False)