Dispatch throws exceptions, Ning doesn't catch… neither does Netty!

[Daniel Spiewak]

I've been meaning to report this for a while, but I keep forgetting.

[error]    dispatch.StatusCode: Unexpected response status: 404 (NettyResponseFuture.java:342)
[error] com.ning.http.client.providers.netty.NettyResponseFuture.abort(NettyResponseFuture.java:342)
[error] com.ning.http.client.providers.netty.NettyAsyncHttpProvider.abort(NettyAsyncHttpProvider.java:1418)
[error] com.ning.http.client.providers.netty.NettyAsyncHttpProvider$HttpProtocol.handle(NettyAsyncHttpProvider.java:2257)
[error] com.ning.http.client.providers.netty.NettyAsyncHttpProvider.messageReceived(NettyAsyncHttpProvider.java:1228)
[error] org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
[error] org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
[error] org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
[error] org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142)
[error] org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
[error] org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
[error] org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
[error] org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
[error] org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
[error] org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
[error] org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
[error] org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92)
[error] org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
[error] org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
[error] org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
[error] org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
[error] org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
[error] org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
[error] org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
[error] org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
[error] org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
[error] org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
[error] org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
[error] dispatch.OkHandler$class.onStatusReceived(handlers.scala:37)
[error] dispatch.OkFunctionHandler.onStatusReceived(handlers.scala:29)
[error] com.ning.http.client.providers.netty.NettyAsyncHttpProvider.updateStatusAndInterrupt(NettyAsyncHttpProvider.java:1560)
[error] com.ning.http.client.providers.netty.NettyAsyncHttpProvider.access$2600(NettyAsyncHttpProvider.java:161)
[error] com.ning.http.client.providers.netty.NettyAsyncHttpProvider$HttpProtocol.handle(NettyAsyncHttpProvider.java:2212)
[error] com.ning.http.client.providers.netty.NettyAsyncHttpProvider.messageReceived(NettyAsyncHttpProvider.java:1228)
[error] org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142)
[error] org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
[error] org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
[error] org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
[error] org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
[error] org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92)
[error] org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
[error] org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
[error] org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
[error] org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
[error] org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
[error] org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
[error] org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)

This is obviously just a stock 404 response. The problem is that it's turned into an exception which KILLS A NETTY POOL THREAD!!!! Dispatch appears to be making the assumption that either AsyncHttp or Netty will catch exceptions thrown in handler threads and convert said exceptions into failures within the Netty future. This does not happen. Netty requires future failure to be signaled explicitly, and AsyncHttpClient makes no effort to perform any such signaling.

Thus, any time you receive an http response with Dispatch that has a 400- or 500-derived status code, you will lose a Netty pool thread and get an ugly stack trace printed to your stderr.

[Matt Farmer]

Sorry it took so long to get around to this, but as mentioned in http4s/http4s#503 the underlying bug here was with AHC, and it seems to have been corrected in recent releases. I've just re-confirmed that the thread pool doesn't get killed locally using a method similar to the one that the maintainer of http4s used to test it on their end.

Closing this issue accordingly, but feel free to write back if you feel there are unaddressed concerns.