Detect Timeout Pages
rotari opened this issue · comments
Radu Rotari commented
Is there a way to detect which pages have time out?
There is a config to set up the time after which times out but I haven't find out a way to catch the pages that never responded.
Mike Chen commented
When timedout
, crawler itself retries in terms of the configuration, failed until retry times use up. Thus you may detect it in callback of the current task, to see if a timedout
error occurred or not, finally handle it.