Add request timeout
marzique opened this issue · comments
I'm scraping a big list of urls, and for each website I use favicon
to extract images that I then store in database. And only when new website gets added to the list - it performs get request:
icons = favicon.get("http://" + value)
return icons[0].url
The problem is that some laggy websites may delay page load for minutes, and all I want is to somehow limit time of each request.
For example :
if images not downloaded after 5 seconds:
return None
Thanks.
Yes, I can add the parameter timeout
to favicon.get()
. I will try to do it on Sunday or feel free to submit a pull request.
https://2.python-requests.org/en/master/user/quickstart/#timeouts
Upgrade to favicon===0.7.0
and you can pass timeout
as a keyword argument: favicon.get(url, timeout=2)
to handle slow sites. See more information on timeout
at https://2.python-requests.org/en/master/user/quickstart/#timeouts