Implement fast failure
olamy opened this issue · comments
The generator doesn't fail fast and keep trying to send requests whereas httpclients has been stopped.
From logs
First error
2017-03-29 05:52:06.689:DBUG:omjlg.LoadGenerator:HttpClient@714446869-scheduler: completed Resource@7df0a336{GET /js/bootstrap.js - 0/0}: Result[HttpRequest[GET /js/bootstrap.js HTTP/1.1]@53d0aa0f > HttpResponse[null 0 null]@5d174e05] java.net.SocketException: Could not connect to load-generator-app-dot-jetty9-work.appspot.com/2607:f8b0:4000:80f:0:0:0:2014:443
2017-03-29 05:52:06.689:DBUG:omjlg.LoadGenerator:HttpClient@714446869-scheduler: failed tree for Resource@5547df0f{GET index.html - 0/0}
2017-03-29 05:52:06.689:DBUG:omjlg.LoadGenerator:HttpClient@714446869-scheduler: sender thread pool-5-thread-1 failed
2017-03-29 05:52:06.690:DBUG:omjlg.LoadGenerator:pool-5-thread-2: stopping http clients
2017-03-29 05:52:06.712:DBUG:omjlg.LoadGenerator:pool-5-thread-2: sender thread pool-5-thread-1 failed
2017-03-29 05:52:06.712:DBUG:omjlg.LoadGenerator:pool-5-thread-2: completed Resource@7df0a336{GET /js/bootstrap.js - 0/0}: Result[HttpRequest[GET /js/bootstrap.js HTTP/1.1]@715e74f4 > HttpResponse[HTTP/1.1 200 OK]@2175ae87] java.nio.channels.AsynchronousCloseException
Then the loadgenerator keep trying to send requests which fail:
2017-03-29 06:08:28.111:WARN:omjlgl.QpsListenerDisplay:pool-5-thread-1: failure to send request: class java.util.concurrent.RejectedExecutionException : org.eclipse.jetty.client.HttpClient@5a862a08 is stopped
2017-03-29 06:08:28.111:DBUG:omjlg.LoadGenerator:pool-5-thread-1: completed Resource@1d138e05{GET /js/bootstrap.js - 0/0}: Result[HttpRequest[GET /js/bootstrap.js HTTP/1.1]@62c59496 > HttpResponse[null 0 null]@283b08f] java.util.concurrent.RejectedExecutionException: org.eclipse.jetty.client.HttpClient@5a862a08 is stopped
2017-03-29 06:08:28.111:DBUG:omjlg.LoadGenerator:pool-5-thread-1: sending Resource@4c47d96b{GET /js/bootstrap.js - 0/0}
2017-03-29 06:08:28.111:WARN:omjlgl.QpsListenerDisplay:pool-5-thread-1: failure to send request: class java.util.concurrent.RejectedExecutionException : org.eclipse.jetty.client.HttpClient@5a862a08 is stopped
2017-03-29 06:08:28.111:DBUG:omjlg.LoadGenerator:pool-5-
The load generator should finish the requests and then stop the HttpClient.
How do you reproduce this issue ?
see #14
I wonder if we could add a configuration to not fail fast (and not stop httpclient) maybe users will be happy to continue load testing after timeout
WDYT?
I'm not sure what is this about.
If there is a failure in Jenkins, we should understand why it fails.
Adding a fail-fast does not seem the right thing to do, at least not in the context of this issue.
Why is the Jenkins test failing ?
My main concern is we keep trying to send request whereas httpclients has been closed. So if the load has been configured to run 30 minutes and something fail after 5 minutes, the generator keep trying sending requests and all are rejected. See the unit test. This can happen outside of Jenkins.
Maybe we can add a boolean parameter fastFail? Or better report problems? (see #12 )
Let's have a chat :-)
@olamy, ok so if there is a failure, the callback passed to sendResourceTree() is failed.
This in turn fails the CompletableFuture created in LoadGenerator.process().
IIUC, you want that on failure we exit the while loop in process(), so it should be enough to change:
if (lastIteration || ranEnough) {
break;
}into
if (lastIteration || ranEnough || process.isCompletedExceptionally()) {
break;
}WDYT ?
@sbordet
yes look good to me! (I have updated the pr).
I wonder if we could make this configurable? use case: running load test, server unavailble for 2/3 minutes but back then continue running the tests
@olamy not sure we want to cover such case... this library is a load generator, and if the server goes down we don't really measure anything, so I would leave that case out.
@sbordet agree this make sense. So I will merge the pr as it (after conflict merge)
merged