Cannot cast java AtomicLong exception and custom Concurrent connections is leaky over time and

Question

Cannot cast java AtomicLong exception and custom Concurrent connections is leaky over time and

AkashKumarAM21 opened this issue 8 months ago · comments

We have two issue, which we think could be related.

Netty infrequently throws -

encountered due to root cause: java.lang.ClassCastException: class reactor.netty.http.server.HttpServerOperations cannot be cast to class java.util.concurrent.atomic.AtomicLong (reactor.netty.http.server.HttpServerOperations is in unnamed module of loader org.springframework.boot.loader.LaunchedURLClassLoader @59e84876; java.util.concurrent.atomic.AtomicLong is in module java.base of loader 'bootstrap')
com.xxxxx.xxx.xxx: Unrecognized Exception: Unexpected exception from Netty
      at com.xxxxx.xxxxx(xxxxx.java:196) ~[classes!/:?]
      at com.xxxxx.xxxxx(xxxxx.java:101) ~[classes!/:?]
      at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
      at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.RuntimeException: Unexpected exception from Netty
      at com.xxxxx.xxxxx(xxxxx.java:87) ~[classes!/:?]
      ... 6 more
Caused by: java.lang.ClassCastException: class reactor.netty.http.server.HttpServerOperations cannot be cast to class java.util.concurrent.atomic.AtomicLong (reactor.netty.http.server.HttpServerOperations is in unnamed module of loader org.springframework.boot.loader.LaunchedURLClassLoader @59e84876; java.util.concurrent.atomic.AtomicLong is in module java.base of loader 'bootstrap')
      at reactor.netty.http.server.HttpServerOperations.requestsCounter(HttpServerOperations.java:756) ~[reactor-netty-http-1.0.27.jar!/:1.0.27]
      at reactor.netty....<TRUNCATED>...5) ~[reactor-netty-http-1.0.27.jar!/:1.0.27]
      at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717) ~[Netty4-22-4.1.jar!/:4.1.78.Final]
      at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709) ~[Netty4-22-4.1.jar!/:4.1.78.Final]
      at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792) ~[Netty4-22-4.1.jar!/:4.1.78.Final]
      at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702) ~[Netty4-22-4.1.jar!/:4.1.78.Final]
      at reactor.netty.http.server.AbstractHttpServerMetricsHandler.write(AbstractHttpServerMetricsHandler.java:154) ~[reactor-netty-http-1.0.27.jar!/:1.0.27]
      at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717) ~[Netty4-22-4.1.jar!/:4.1.78.Final]
      at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709) ~[Netty4-22-4.1.jar!/:4.1.78.Final]
      at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792) ~[Netty4-22-4.1.jar!/:4.1.78.Final]
      at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702) ~[Netty4-22-4.1.jar!/:4.1.78.Final]
      at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:110) ~[Netty4-2-4.1.jar!/:4.1.78.Final]
      at io.netty.handler.codec.MessageToMessageCodec.write(MessageToMessageCodec.java:116) ~[Netty4-2-4.1.jar!/:4.1.78.Final]
      at reactor.netty.http.server.SimpleCompressionHandler.write(SimpleCompressionHandler.java:45) ~[reactor-netty-http-1.0.27.jar!/:1.0.27]
      at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717) ~[Netty4-22-4.1.jar!/:4.1.78.Final]
      at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:764) ~[Netty4-22-4.1.jar!/:4.1.78.Final]
      at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071) ~[Netty4-22-4.1.jar!/:4.1.78.Final]
      at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[Netty4-14-4.1.jar!/:4.1.78.Final]
      at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[Netty4-14-4.1.jar!/:4.1.78.Final]
      at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[Netty4-14-4.1.jar!/:4.1.78.Final]
      at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:391) ~[Netty4-23-4.1.jar!/:4.1.78.Final]
      at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[Netty4-14-4.1.jar!/:4.1.78.Final]
      at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[Netty4-14-4.1.jar!/:4.1.78.Final]
      at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[Netty4-14-4.1.jar!/:4.1.78.Final]
      ... 1 more

We emit our own simple custom concurrent connections metric, that metric over time keeps on growing, only decreases when we reboot the server.

private final AtomicInteger concurrentConnections = new AtomicInteger();
 
public Publisher<Void> handle(HttpServerRequest httpServerRequest, HttpServerResponse httpServerResponse) {
        emitMetric(concurrentConnections.incrementAndGet());
         
        try {
            SubscriberInputStream subscriberInputStream = new SubscriberInputStream();
            try (final CloseableThreadContext.Instance ignored = CloseableThreadContext.put("RequestId", requestId)) {
                try {
                    return rateLimiter.executeSupplier(() -> {
                        return handleRequest();
                    });
                } catch (RequestNotPermitted e) {
                    return handleFailure();
                }
            }
        } catch (Throwable e) {
            return handleFailure();
        }
    }
 
    private Publisher<Void> handleRequest() {
        CompletableFuture<Void> result = CompletableFuture.runAsync(() -> {
            try () {
                // action 
            } finally {
                concurrentConnections.decrementAndGet();
            }
        }, executor);
        return Mono.fromCompletionStage(result);
    }
 
    private Publisher<Void> handleFailure() {
        concurrentConnections.decrementAndGet();
    }

Expected Behavior

We are expecting the concurrentConnections to decrement when we complete handling the request, either successfully or due to error and we are not sure why Netty throws casting exception.

Your Environment

Reactor version(s) used: io.projectreactor_reactor-core-3.4.24
Other relevant libraries versions (eg. netty, ...): Netty 4.1
JVM version (java -version): JDK17
OS and version (eg. uname -a): AL2_x86_64

Violeta Georgieva · Answer 1 · Wed Dec 06 2023 03:10:58 GMT+0800 (China Standard Time)

@AkashKumarAM21 Can you check for two versions of Reactor Netty in your dependencies?

Violeta Georgieva · Answer 2 · Wed Dec 06 2023 03:22:04 GMT+0800 (China Standard Time)

@AkashKumarAM21 Another thing that you might try to check is the change of the class loaders described here spring-projects/spring-boot#19427

AkashKumarAM21 · Answer 3 · Wed Dec 06 2023 05:40:13 GMT+0800 (China Standard Time)

@AkashKumarAM21 Can you check for two versions of Reactor Netty in your dependencies?

@violetagg We can't have duplicate versions of same dependency and we fail if there are duplicate class paths from different packages, our build tools enforce it! I'll double check it though.

AkashKumarAM21 · Answer 4 · Wed Dec 06 2023 08:06:23 GMT+0800 (China Standard Time)

This is the latest stack trace, after we upgraded the dependencies.

encountered due to root cause: java.lang.ClassCastException: class reactor.netty.http.server.HttpServerOperations cannot be cast to class java.util.concurrent.atomic.AtomicLong (reactor.netty.http.server.HttpServerOperations is in unnamed module of loader org.springframework.boot.loader.LaunchedURLClassLoader @39fb3ab6; java.util.concurrent.atomic.AtomicLong is in module java.base of loader 'bootstrap')
	at com.xxxx.xxxxx(xxxx.java:106) ~[classes!/:?]
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: java.lang.RuntimeException: Unexpected exception from Netty
	at com.xxxx.xxxx(xxxx.java:89) ~[classes!/:?]
	at java.io.OutputStream.write(OutputStream.java:162) ~[?:?]
	at java.io.InputStream.transferTo(InputStream.java:783) ~[?:?]
	at com.xxxx.xxxx(xxxx.java:252) ~[classes!/:?]
	... 6 more
	Suppressed: java.lang.RuntimeException: Unexpected exception from Netty
		at com.xxxx.xxxx(xxxxxxx.java:89) ~[classes!/:?]
		at com.xxxx.xxxx(xxxxxxx.java:66) ~[classes!/:?]
		at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
		at java.lang.Thread.run(Thread.java:840) [?:?]
	Caused by: java.lang.ClassCastException: class reactor.netty.http.server.HttpServerOperations cannot be cast to class java.util.concurrent.atomic.AtomicLong (reactor.netty.http.server.HttpServerOperations is in unnamed module of loader org.springframework.boot.loader.LaunchedURLClassLoader @39fb3ab6; java.util.concurrent.atomic.AtomicLong is in module java.base of loader 'bootstrap')
		at reactor.netty.http.server.HttpServerOperations.requestsCounter(HttpServerOperations.java:788) ~[reactor-netty-http-1.0.36.jar!/:1.0.36]
		at reactor.netty.http.server.HttpTrafficHandler.write(HttpTrafficHandler.java:307) ~[reactor-netty-http-1.0.36.jar!/:1.0.36]
		at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:879) ~[Netty4-23-4.1.jar!/:4.1.100.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:863) ~[Netty4-23-4.1.jar!/:4.1.100.Final]
		at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:968) ~[Netty4-23-4.1.jar!/:4.1.100.Final]
		at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:856) ~[Netty4-23-4.1.jar!/:4.1.100.Final]
		at reactor.netty.http.server.AbstractHttpServerMetricsHandler.write(AbstractHttpServerMetricsHandler.java:154) ~[reactor-netty-http-1.0.36.jar!/:1.0.36]
		at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:879) ~[Netty4-23-4.1.jar!/:4.1.100.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:863) ~[Netty4-23-4.1.jar!/:4.1.100.Final]
		at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:968) ~[Netty4-23-4.1.jar!/:4.1.100.Final]
		at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:856) ~[Netty4-23-4.1.jar!/:4.1.100.Final]
		at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:113) ~[Netty4-2-4.1.jar!/:4.1.100.Final]
		at io.netty.handler.codec.MessageToMessageCodec.write(MessageToMessageCodec.java:116) ~[Netty4-2-4.1.jar!/:4.1.100.Final]
		at reactor.netty.http.server.SimpleCompressionHandler.write(SimpleCompressionHandler.java:50) ~[reactor-netty-http-1.0.36.jar!/:1.0.36]
		at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:879) ~[Netty4-23-4.1.jar!/:4.1.100.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:940) ~[Netty4-23-4.1.jar!/:4.1.100.Final]
		at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1247) ~[Netty4-23-4.1.jar!/:4.1.100.Final]
		at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[Netty4-14-4.1.jar!/:4.1.100.Final]
		at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[Netty4-14-4.1.jar!/:4.1.100.Final]
		at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[Netty4-14-4.1.jar!/:4.1.100.Final]
		at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:416) ~[Netty4-24-4.1.jar!/:4.1.100.Final]
		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[Netty4-14-4.1.jar!/:4.1.100.Final]
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[Netty4-14-4.1.jar!/:4.1.100.Final]
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[Netty4-14-4.1.jar!/:4.1.100.Final]
		... 1 more
Caused by: java.lang.ClassCastException: class reactor.netty.http.server.HttpServerOperations cannot be cast to class java.util.concurrent.atomic.AtomicLong (reactor.netty.http.server.HttpServerOperations is in unnamed module of loader org.springframework.boot.loader.LaunchedURLClassLoader @39fb3ab6; java.util.concurrent.atomic.AtomicLong is in module java.base of loader 'bootstrap')
	at reactor.netty.http.server.HttpServerOperations.requestsCounter(HttpServerOperations.java:788) ~[reactor-netty-http-1.0.36.jar!/:1.0.36]
	at reactor.netty.http.server.HttpTrafficHandler.write(HttpTrafficHandler.java:307) ~[reactor-netty-http-1.0.36.jar!/:1.0.36]
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:879) ~[Netty4-23-4.1.jar!/:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:863) ~[Netty4-23-4.1.jar!/:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:968) ~[Netty4-23-4.1.jar!/:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:856) ~[Netty4-23-4.1.jar!/:4.1.100.Final]
	at reactor.netty.http.server.AbstractHttpServerMetricsHandler.write(AbstractHttpServerMetricsHandler.java:154) ~[reactor-netty-http-1.0.36.jar!/:1.0.36]
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:879) ~[Netty4-23-4.1.jar!/:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:863) ~[Netty4-23-4.1.jar!/:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:968) ~[Netty4-23-4.1.jar!/:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:856) ~[Netty4-23-4.1.jar!/:4.1.100.Final]
	at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:113) ~[Netty4-2-4.1.jar!/:4.1.100.Final]
	at io.netty.handler.codec.MessageToMessageCodec.write(MessageToMessageCodec.java:116) ~[Netty4-2-4.1.jar!/:4.1.100.Final]
	at reactor.netty.http.server.SimpleCompressionHandler.write(SimpleCompressionHandler.java:50) ~[reactor-netty-http-1.0.36.jar!/:1.0.36]
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:879) ~[Netty4-23-4.1.jar!/:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:940) ~[Netty4-23-4.1.jar!/:4.1.100.Final]
	at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1247) ~[Netty4-23-4.1.jar!/:4.1.100.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[Netty4-14-4.1.jar!/:4.1.100.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[Netty4-14-4.1.jar!/:4.1.100.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[Netty4-14-4.1.jar!/:4.1.100.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:416) ~[Netty4-24-4.1.jar!/:4.1.100.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[Netty4-14-4.1.jar!/:4.1.100.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[Netty4-14-4.1.jar!/:4.1.100.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[Netty4-14-4.1.jar!/:4.1.100.Final]
	... 1 more
	```

AkashKumarAM21 · Answer 5 · Thu Dec 07 2023 02:03:11 GMT+0800 (China Standard Time)

@AkashKumarAM21 Another thing that you might try to check is the change of the class loaders described here spring-projects/spring-boot#19427

@violetagg We already provide custom Executor for our runAsync call, so this is not the issue ?

Violeta Georgieva · Answer 6 · Thu Dec 07 2023 16:40:42 GMT+0800 (China Standard Time)

@AkashKumarAM21 Can you provide a reproducible example so that we can investigate the issue?

AkashKumarAM21 · Answer 7 · Fri Dec 08 2023 04:36:46 GMT+0800 (China Standard Time)

@AkashKumarAM21 Can you provide a reproducible example so that we can investigate the issue?

Thats where we came to a blocker, this exception is infrequently (like 1 request fails for every 200,000 requests) and we are unable to produce it. We tried upgrading our dependencies to check if thats going to fix the issue, not luck yet.

Violeta Georgieva · Answer 8 · Thu Dec 21 2023 00:38:45 GMT+0800 (China Standard Time)

@AkashKumarAM21 It is hard for us to proceed without a clear scenario/reproducible example.

AkashKumarAM21 · Answer 9 · Fri Dec 29 2023 01:17:47 GMT+0800 (China Standard Time)

@AkashKumarAM21 It is hard for us to proceed without a clear scenario/reproducible example.

Yup, i understand that! we don't have any reproduction steps :(

Is there any feature/property that we can enable to get more information ? It would be helpfully if there is any logging or instrumentation we can turn on to get more details in this scenario.

Violeta Georgieva · Answer 10 · Fri Jan 05 2024 21:24:01 GMT+0800 (China Standard Time)

@AkashKumarAM21 Reactor Netty never changes the class loaders, we don't have any logging about this.

Violeta Georgieva · Answer 11 · Mon Jan 15 2024 16:41:44 GMT+0800 (China Standard Time)

@AkashKumarAM21 I'm closing this as I cannot reproduce the issue. We can reopen it, if you are able to provide reproducible example.