IllegalStateException with async REST service
nezihyigitbasi opened this issue · comments
I am seeing the below IllegalStateExceptions in our integration tests with an async REST service and so far we don't have any reliable reproduction.
java.lang.IllegalStateException: s=DISPATCHED i=false a=NOT_ASYNC
at org.eclipse.jetty.server.HttpChannelState.complete(HttpChannelState.java:681)
at org.eclipse.jetty.server.AsyncContextState.complete(AsyncContextState.java:99)
at org.glassfish.jersey.servlet.async.AsyncContextDelegateProviderImpl$ExtensionImpl.complete(AsyncContextDelegateProviderImpl.java:125)
at org.glassfish.jersey.servlet.internal.ResponseWriter.commit(ResponseWriter.java:197)
at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:413)
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:784)
at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$3.run(ServerRuntime.java:934)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:966)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:922)
at com.facebook.presto.server.TestingAsyncResponseHandler$1.onSuccess(TaskResource.java:408)
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1135)
at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2017-10-03 05:13:16 WARNING Attempt to release request processing resources has failed for a request.
java.lang.IllegalStateException: AsyncContext completed and/or Request lifecycle recycled
at org.eclipse.jetty.server.AsyncContextState.state(AsyncContextState.java:54)
at org.eclipse.jetty.server.AsyncContextState.complete(AsyncContextState.java:99)
at org.glassfish.jersey.servlet.async.AsyncContextDelegateProviderImpl$ExtensionImpl.complete(AsyncContextDelegateProviderImpl.java:125)
at org.glassfish.jersey.servlet.internal.ResponseWriter.commit(ResponseWriter.java:197)
at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:413)
at org.glassfish.jersey.server.ServerRuntime$Responder.release(ServerRuntime.java:810)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:435)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$3.run(ServerRuntime.java:934)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:966)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:922)
at com.facebook.presto.server.TestingAsyncResponseHandler$1.onSuccess(TaskResource.java:408)
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1135)
at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
The REST service code is here, and the class where we do the async resume call is here (in these stack traces TestingAsyncResponseHandler resumes the request, but it's the exact same code with AsyncResponseHandler). We are using Jersey 2.22.2 and Jetty 9.4.7.v20170914.
So far I have been investigating this issue with the Jetty team, at the same time I also wanted to get the opinions of Jersey experts and see whether this can be a Jersey issue. Any input is greatly appreciated.
The Jetty's team analysis of this issue is that we have detected an error while the request is asynchronously waiting (probably a HTTP/2 stream reset - which is a semantic that doesn't exit in HTTP/1 which cannot detect such errors while waiting). We attempt to deliver the error by calling any AsyncListener#onError callbacks, but there are none. Then as per the servlet specification, the async cycle is ended and we perform an non-async error dispatch.
It is apparent that jersey does not see the onError callback (as it has no listener for that) and continues to try to produce the response itself. This will probably result in racy non deterministic behaviour with the calls on the Response object (which is not thread safe) and then an ISE when AsyncContext is finally called.
We believe that Jersey should register an AsyncListener with onError handling that will atomically cancel and/or render a noop your async resumption.
@pavelbucek @mpotociar any ideas about this issue?
Tested with Jersey 2.26 and still see the same exceptions with different stacks:
2017-10-11 05:46:40 WARNING Attempt to release request processing resources has failed for a request.
java.lang.IllegalStateException: AsyncContext completed and/or Request lifecycle recycled
at org.eclipse.jetty.server.AsyncContextState.state(AsyncContextState.java:54)
at org.eclipse.jetty.server.AsyncContextState.complete(AsyncContextState.java:99)
at org.glassfish.jersey.servlet.async.AsyncContextDelegateProviderImpl$ExtensionImpl.complete(AsyncContextDelegateProviderImpl.java:126)
at org.glassfish.jersey.servlet.internal.ResponseWriter.commit(ResponseWriter.java:197)
at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:414)
at org.glassfish.jersey.server.ServerRuntime$Responder.release(ServerRuntime.java:760)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:386)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$3.run(ServerRuntime.java:884)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:916)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:872)
at io.airlift.http.server.AsyncResponseHandler$1.onSuccess(AsyncResponseHandler.java:102)
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1135)
at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2017-10-11 05:46:40 SEVERE Error while closing the output stream in order to commit response.
java.lang.IllegalStateException: s=DISPATCHED i=false a=NOT_ASYNC
at org.eclipse.jetty.server.HttpChannelState.complete(HttpChannelState.java:681)
at org.eclipse.jetty.server.AsyncContextState.complete(AsyncContextState.java:99)
at org.glassfish.jersey.servlet.async.AsyncContextDelegateProviderImpl$ExtensionImpl.complete(AsyncContextDelegateProviderImpl.java:126)
at org.glassfish.jersey.servlet.internal.ResponseWriter.commit(ResponseWriter.java:197)
at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:414)
at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:734)
at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:395)
at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:385)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$3.run(ServerRuntime.java:884)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:916)
at org.glassfish.jersey.server.ServerRuntime$AsyncResponder.resume(ServerRuntime.java:872)
at io.airlift.http.server.AsyncResponseHandler$1.onSuccess(AsyncResponseHandler.java:102)
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1135)
at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
I've got the same problem. We can reproduce easily using jetty h2/h2c connectors and the h2load tool (https://nghttp2.org/documentation/h2load-howto.html).
Just run some long running load tests using h2load on some @Suspended AsyncResponse route, then hit ctrl-c on h2load and you should see hundreds of error logs like the one mentioned here.
Any solution on this? We are getting this error. Will resources may not be release in this case? Will it lead to too many open files error?
@plelevier @aphalke Jetty has implemented a work around, which is in our current 9.4.9 release. See issue jetty/jetty.project#1891 and commit jetty/jetty.project@4236f14
It is a configuration setter in HttpConfiguration.