[FEATURE] Add a parameter to configure the thereshold for response time
onlinebizsoft opened this issue · comments
Is your feature request related to a problem?
Sometime the server may get a problem during crawling, the crawler should stop
Describe the solution you'd like
Add a threshold parameter to stop the crawler when response time is over that threshold
Describe alternatives you've considered
No response
Additional context
No response
This should already be doable with Guzzle's timeout request option. You can pass request options with the --crawler-options command option. The only thing left is that the crawler won't stop on failures, e.g. timeouts. I may add an option --stop-on-failure or similar later, so this issue will be kept open.
@eliashaeussler yes we need the script to stop because when that happen means the server may get overloading or some problems so we shouldn't continue
@onlinebizsoft I've implemented a new command option --stop-on-failure. You can use this in combination with what I wrote in #301 (comment). This is released with version 2.7.0 of the library.