Amazon CloudFront shouldn't be treated as crawler
sunnydavis opened this issue · comments
Amazon CloudFront is often the user agent for using CloudFront as reverse proxy / CDN so it's actually not a crawler but regular user requests.
Fancy PRing that?
Thanks 👍🏻
Sure, it's done. #410
Merged 👍