bda-research / node-crawler

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

Home Page:http://node-crawler.org

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Detect Timeout Pages

rotari opened this issue · comments

Is there a way to detect which pages have time out?

There is a config to set up the time after which times out but I haven't find out a way to catch the pages that never responded.

When timedout, crawler itself retries in terms of the configuration, failed until retry times use up. Thus you may detect it in callback of the current task, to see if a timedout error occurred or not, finally handle it.