Crawl-Delay not supported

[Troels Knak-Nielsen]

Hi,

Nice class. I doesn't recognise the directive for Crawl-Delay though. (See: http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive and https://ahrefs.com/robot/)

[Igor Timoshenkov]

Hi. Thanks for the suggestion.

I will look over this and update the code shortly. Alternatively - you could submit a pull-request.

Thanks,
Igor

[Igor Timoshenkov]

It would also be very helpful if you could provide an example of the file that you are trying to parse, to know what should we expect.

[Troels Knak-Nielsen]

The ahrefs link shows an example. I'll have a look at making a pull request.

[Troels Knak-Nielsen]

Opened pull request #5