abrenaut / waybackscraper

Scrapes a website archives using Python's asyncio and aiohttp.

Please be nicer to the Archive

wumpus opened this issue 8 years ago · comments

Greg Lindahl commented 8 years ago

Please use a useragent which indicates that it's waybackscraper
Please lower the default concurrency, 10 is a lot.

Arthur Brenaut commented 8 years ago

I set up a user agent and lowered the default concurrency to 2.

Thanks for the feedback