Reactor Netty IllegalReferenceCountException: refCnt: 0

Question

Reactor Netty IllegalReferenceCountException: refCnt: 0

528216045 opened this issue 7 months ago · comments

Netty byte buffer release bug occurred in SpringCloudGateway project.
We develop and run an API Gateway program based on SpringCloudGateway framework，which accept requests from jmeter and forward to backend micro service。When some request timeout between API Gateway and backend micro service ，the IllegalReferenceCountException Exception occurred in other normal request. The two Error Stack as follows

netty byte buffer repeat release

netty byte was already released in being used

Expected Behavior

When some request timeout between API Gateway and backend micro service ，the other request should not be effected。 So the IllegalReferenceCountException exception should not be appeared.

Actual Behavior

netty byte buffer repeat release(IllegalReferenceCountException: refCnt: 0, decrement:1)
netty byte was already released in being used(IllegalReferenceCountException: refCnt: 0)

Steps to Reproduce

Create Java apigateway project based on SpringCloudGateway and micro service based on SpringBoot.
Simulate some request timeout between backend micro service and apigateway(timeout exception occurred in apigateway. eg.. we use Thread.sleep in micro service's specific api)
the other normal request will appear IllegalReferenceCountException in apigateway

Possible Solution

This issue is related to this bug .spring-projects/spring-framework#26060

Your Environment

Reactor version(s) used:
Other relevant libraries versions (eg. netty, ...): netty 4.1.101.Final
Spring Framework: 5.3.31
Spring clouds gateway:3.1.9
JVM version (java -version): 1.8
OS and version (eg. uname -a):

Violeta Georgieva · Answer 1 · Fri Mar 08 2024 20:21:53 GMT+0800 (China Standard Time)

I'm gonna check this one.

@528216045 If you already have the reproducible example, can you follow this FAQ and provide the extended logs?
https://projectreactor.io/docs/netty/release/reference/index.html#faq.memory-leaks

Lennart · Answer 2 · Thu Mar 28 2024 21:16:59 GMT+0800 (China Standard Time)

Hey @violetagg, thanks for responding to this issue. I've got the same issues and can attach details to perhaps speed up the investigation.

However, I run a lightly different constellation of versions. I'm running:
Spring Boot v3.0.13
spring-web 6.0.14
Netty: 4.1.101.Final
Reactor-Netty: 1.1.13

Hope this helps.

2024-03-28 14:05:46.548 CET ERROR requestId=NA x-correlation-id=NA 6469 --- [actor-tcp-nio-4] io.netty.util.ResourceLeakDetector       : LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
#1:
	io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:617)
	org.springframework.core.io.buffer.NettyDataBuffer.write(NettyDataBuffer.java:181)
	org.springframework.core.io.buffer.NettyDataBuffer.write(NettyDataBuffer.java:40)
	my.application.core.manager.snapshot.reader.SnapshotArchiveReader.readEntryBytes(SnapshotArchiveReader.java:213)
	my.application.core.manager.snapshot.reader.SnapshotArchiveReader.lambda$createSink$7(SnapshotArchiveReader.java:134)
	reactor.core.publisher.FluxCreate$BaseSink.request(FluxCreate.java:530)
	reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.request(FluxOnAssembly.java:649)
	reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.request(FluxContextWrite.java:136)
	reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.request(FluxOnAssembly.java:649)
	reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.requestUpstream(FluxSubscribeOn.java:131)
	reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.request(FluxSubscribeOn.java:175)
	reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.request(FluxOnAssembly.java:649)
	reactor.core.publisher.FluxWindowPredicate$WindowFlux.drainRegular(FluxWindowPredicate.java:684)
	reactor.core.publisher.FluxWindowPredicate$WindowFlux.drain(FluxWindowPredicate.java:748)
	reactor.core.publisher.FluxWindowPredicate$WindowFlux.subscribe(FluxWindowPredicate.java:823)
	reactor.core.publisher.Flux.subscribe(Flux.java:8773)
	reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:427)
	reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onNext(FluxOnAssembly.java:539)
	reactor.core.publisher.FluxSkip$SkipSubscriber.onNext(FluxSkip.java:87)
	reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onNext(FluxOnAssembly.java:539)
	reactor.core.publisher.FluxWindowPredicate$WindowPredicateMain.drainLoop(FluxWindowPredicate.java:496)
	reactor.core.publisher.FluxWindowPredicate$WindowPredicateMain.drain(FluxWindowPredicate.java:432)
	reactor.core.publisher.FluxWindowPredicate$WindowPredicateMain.onNext(FluxWindowPredicate.java:253)
	reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onNext(FluxOnAssembly.java:539)
	reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.onNext(FluxSubscribeOn.java:151)
	reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onNext(FluxOnAssembly.java:539)
	reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
	reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onNext(FluxOnAssembly.java:539)
	reactor.core.publisher.FluxCreate$BufferAsyncSink.drain(FluxCreate.java:878)
	reactor.core.publisher.FluxCreate$BufferAsyncSink.next(FluxCreate.java:803)
	reactor.core.publisher.FluxCreate$SerializedFluxSink.drainLoop(FluxCreate.java:256)
	reactor.core.publisher.FluxCreate$SerializedFluxSink.next(FluxCreate.java:176)
	my.application.core.manager.snapshot.reader.SnapshotArchiveReader.lambda$createSink$7(SnapshotArchiveReader.java:146)
	reactor.core.publisher.FluxCreate$BaseSink.onPushPullRequest(FluxCreate.java:572)
	reactor.core.publisher.FluxCreate$SerializedFluxSink.onRequest(FluxCreate.java:271)
	my.application.core.manager.snapshot.reader.SnapshotArchiveReader.lambda$createSink$8(SnapshotArchiveReader.java:129)
	reactor.core.publisher.FluxCreate.subscribe(FluxCreate.java:95)
	reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62)
	reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194)
	reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)
	reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)
	java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:317)
	java.base/java.util.concurrent.FutureTask.run(FutureTask.java)
	java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	java.base/java.lang.Thread.run(Thread.java:1583)
Created at:
	io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:402)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
	io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:116)
	org.springframework.core.io.buffer.NettyDataBufferFactory.allocateBuffer(NettyDataBufferFactory.java:72)
	org.springframework.core.io.buffer.NettyDataBufferFactory.allocateBuffer(NettyDataBufferFactory.java:39)
	my.application.core.manager.snapshot.reader.SnapshotArchiveReader.readEntryBytes(SnapshotArchiveReader.java:212)
	my.application.core.manager.snapshot.reader.SnapshotArchiveReader.lambda$createSink$7(SnapshotArchiveReader.java:134)
	reactor.core.publisher.FluxCreate$BaseSink.request(FluxCreate.java:530)
	reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.request(FluxOnAssembly.java:649)
	reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.request(FluxContextWrite.java:136)
	reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.request(FluxOnAssembly.java:649)
	reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.requestUpstream(FluxSubscribeOn.java:131)
	reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.request(FluxSubscribeOn.java:175)
	reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.request(FluxOnAssembly.java:649)
	reactor.core.publisher.FluxWindowPredicate$WindowFlux.drainRegular(FluxWindowPredicate.java:684)
... (more in leak-logger.txt)

leak-logger.txt

Violeta Georgieva · Answer 3 · Thu Mar 28 2024 21:51:02 GMT+0800 (China Standard Time)

@lennartblom I don't think your use case is the same. I do not see reactor.netty.* packages at all in your stack traces.

Lennart · Answer 4 · Thu Mar 28 2024 23:00:30 GMT+0800 (China Standard Time)

Hi @violetagg. Thanks for following up so quickly.
I'm afraid my logs were a bit incomplete. I focused on the data leak logs. Sorry about that.

I believe this stack trace is a bit more fitting to the initiale issue description, than the data leak logs.

caused by: org.springframework.web.reactive.function.client.WebClientRequestException: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
	caused by: io.netty.handler.codec.EncoderException: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
	caused by: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
	at my.application.core.commons.exception.upstream.ExternalServiceUnavailableException.serviceUnavailableException(ExternalServiceUnavailableException.java:24)
	at my.application.core.upstream.ExternalServiceReactiveExceptionHandler.handleException(ExternalServiceReactiveExceptionHandler.java:141)
	at my.application.core.upstream.ExternalServiceReactiveExceptionHandler.createBlobFromBufferedBytes(ExternalServiceReactiveExceptionHandler.java:91)
	at my.application.core.upstream.ExternalServiceReactiveService.handleError(ExternalServiceReactiveService.java:519)
	at my.application.core.upstream.ExternalServiceReactiveService.lambda$createBlobFromBufferedBytes$42(ExternalServiceReactiveService.java:439)
	at reactor.core.publisher.Mono.lambda$onErrorMap$28(Mono.java:3783)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
	at reactor.core.publisher.FluxMap$MapSubscriber.onError(FluxMap.java:134)
	at reactor.core.publisher.FluxDoOnEach$DoOnEachSubscriber.onError(FluxDoOnEach.java:195)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106)
	at reactor.core.publisher.Operators.error(Operators.java:198)
	at reactor.core.publisher.MonoError.subscribe(MonoError.java:53)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4495)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:180)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121)
	at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onError(FluxDoFinally.java:119)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258)
	at reactor.core.publisher.FluxPeekFuseable$PeekConditionalSubscriber.onError(FluxPeekFuseable.java:903)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onError(Operators.java:2210)
	at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:544)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:241)
	at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:315)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:241)
	at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:315)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121)
	at io.github.resilience4j.reactor.circuitbreaker.operator.CircuitBreakerSubscriber.hookOnError(CircuitBreakerSubscriber.java:86)
	at reactor.core.publisher.BaseSubscriber.onError(BaseSubscriber.java:180)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:180)
	at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222)
	at reactor.core.publisher.FluxDoOnEach$DoOnEachSubscriber.onError(FluxDoOnEach.java:195)
	at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222)
	at reactor.core.publisher.FluxMap$MapSubscriber.onError(FluxMap.java:134)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106)
	at reactor.core.publisher.Operators.error(Operators.java:198)
	at reactor.core.publisher.MonoErrorSupplied.subscribe(MonoErrorSupplied.java:56)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4495)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103)
	at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222)
	at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222)
	at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222)
	at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:93)
	at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onError(MonoFlatMapMany.java:204)
	at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124)
	at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.whenError(FluxRetryWhen.java:225)
	at reactor.core.publisher.FluxRetryWhen$RetryWhenOtherSubscriber.onError(FluxRetryWhen.java:274)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121)
	at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.maybeOnError(FluxConcatMapNoPrefetch.java:326)
	at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.onNext(FluxConcatMapNoPrefetch.java:211)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
	at reactor.core.publisher.SinkManyEmitterProcessor.drain(SinkManyEmitterProcessor.java:471)
	at reactor.core.publisher.SinkManyEmitterProcessor$EmitterInner.drainParent(SinkManyEmitterProcessor.java:615)
	at reactor.core.publisher.FluxPublish$PubSubInner.request(FluxPublish.java:873)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.request(FluxContextWrite.java:136)
	at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.request(FluxConcatMapNoPrefetch.java:336)
	at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.request(FluxContextWrite.java:136)
	at reactor.core.publisher.Operators$DeferredSubscription.request(Operators.java:1717)
	at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onError(FluxRetryWhen.java:192)
	at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:201)
	at reactor.netty.http.client.HttpClientConnect$HttpObserver.onUncaughtException(HttpClientConnect.java:403)
	at reactor.netty.ReactorNetty$CompositeConnectionObserver.onUncaughtException(ReactorNetty.java:703)
	at reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.onUncaughtException(DefaultPooledConnectionProvider.java:213)
	at reactor.netty.resources.DefaultPooledConnectionProvider$PooledConnection.onUncaughtException(DefaultPooledConnectionProvider.java:466)
	at reactor.netty.http.client.HttpClientOperations.onOutboundError(HttpClientOperations.java:649)
	at reactor.netty.channel.ChannelOperations.onError(ChannelOperations.java:263)
	at reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onError(MonoIgnoreElements.java:84)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onError(Operators.java:2210)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onError(MonoIgnoreThen.java:278)
	at reactor.netty.channel.MonoSendMany$SendManyInner.tryFailure(MonoSendMany.java:618)
	at reactor.netty.channel.MonoSendMany$SendManyInner$1.tryFailure(MonoSendMany.java:577)
	at io.netty.util.internal.PromiseNotificationUtil.tryFailure(PromiseNotificationUtil.java:64)
	at io.netty.channel.AbstractChannelHandlerContext.notifyOutboundHandlerException(AbstractChannelHandlerContext.java:990)
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:884)
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:863)
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:968)
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:856)
	at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:304)
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:879)
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:863)
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:968)
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:856)
	at io.netty.handler.timeout.WriteTimeoutHandler.write(WriteTimeoutHandler.java:112)
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:881)
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:863)
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:968)
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:856)
	at reactor.netty.channel.MonoSendMany$SendManyInner.run(MonoSendMany.java:321)
	at reactor.netty.channel.MonoSendMany$SendManyInner.trySchedule(MonoSendMany.java:434)
	at reactor.netty.channel.MonoSendMany$SendManyInner.onNext(MonoSendMany.java:223)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	at reactor.core.publisher.FluxReplay$UnboundedReplayBuffer.replayNormal(FluxReplay.java:618)
	at reactor.core.publisher.FluxReplay$UnboundedReplayBuffer.replay(FluxReplay.java:709)
	at reactor.core.publisher.FluxReplay$ReplayInner.request(FluxReplay.java:1711)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)
	at reactor.netty.channel.MonoSendMany$SendManyInner.onSubscribe(MonoSendMany.java:254)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)
	at reactor.core.publisher.FluxReplay.subscribeOrReturn(FluxReplay.java:1181)
	at reactor.core.publisher.FluxReplay.subscribe(FluxReplay.java:1153)
	at reactor.core.publisher.FluxAutoConnectFuseable.subscribe(FluxAutoConnectFuseable.java:61)
	at reactor.core.publisher.Flux.subscribe(Flux.java:8773)
	at reactor.netty.channel.MonoSendMany.subscribe(MonoSendMany.java:102)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:240)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203)
	at reactor.core.publisher.Operators.complete(Operators.java:137)
	at reactor.netty.FutureMono.doSubscribe(FutureMono.java:122)
	at reactor.netty.FutureMono$DeferredFutureMono.subscribe(FutureMono.java:114)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4495)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263)
	at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4495)
	at reactor.core.publisher.FluxConcatIterable$ConcatIterableSubscriber.onComplete(FluxConcatIterable.java:147)
	at reactor.core.publisher.FluxConcatIterable.subscribe(FluxConcatIterable.java:60)
	at reactor.core.publisher.MonoFromFluxOperator.subscribe(MonoFromFluxOperator.java:81)
	at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53)
	at reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:445)
	at reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:710)
	at reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.run(DefaultPooledConnectionProvider.java:282)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Violeta Georgieva · Answer 5 · Thu Mar 28 2024 23:27:25 GMT+0800 (China Standard Time)

@lennartblom This exception means that the code tries to release a buffer that is released already. The fact that the exception is the same doesn't mean that the cause is the same.

Please provide the code for these custom classes:

	caused by: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
	at my.application.core.commons.exception.upstream.ExternalServiceUnavailableException.serviceUnavailableException(ExternalServiceUnavailableException.java:24)
	at my.application.core.upstream.ExternalServiceReactiveExceptionHandler.handleException(ExternalServiceReactiveExceptionHandler.java:141)
	at my.application.core.upstream.ExternalServiceReactiveExceptionHandler.createBlobFromBufferedBytes(ExternalServiceReactiveExceptionHandler.java:91)
	at my.application.core.upstream.ExternalServiceReactiveService.handleError(ExternalServiceReactiveService.java:519)
	at my.application.core.upstream.ExternalServiceReactiveService.lambda$createBlobFromBufferedBytes$42(ExternalServiceReactiveService.java:439)

Lennart · Answer 6 · Thu Mar 28 2024 23:30:13 GMT+0800 (China Standard Time)

Ah ok. Got it. Thank you very much!! I will discuss with my team, how to proceed further. Perhaps this hint is already giving us enough to mitigate the issue (corporate code base).

Sorry for mixing up the topic.

Lennart · Answer 7 · Mon Apr 08 2024 19:52:17 GMT+0800 (China Standard Time)

Thanks again, @violetagg. Just to update my issue:
We identified some troubling .cache() calls inside our application. This was called previously to some flux operations.

Turns out garbage collection with an spring reactive update got a major update and the previous codebase introduced some hidden .release() call, without realising it.

Thanks for pointing us to the right direction :)

Violeta Georgieva · Answer 8 · Tue Apr 09 2024 20:12:25 GMT+0800 (China Standard Time)

@528216045 I am not able to reproduce the issue. Please either provide a reproducible example that we can run or provide more details following this FAQ https://projectreactor.io/docs/netty/release/reference/index.html#faq.memory-leaks

github-actions · Answer 9 · Wed Apr 17 2024 14:27:43 GMT+0800 (China Standard Time)

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

github-actions · Answer 10 · Wed Apr 24 2024 14:28:01 GMT+0800 (China Standard Time)

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open.

528216045 · Answer 11 · Wed Apr 24 2024 14:39:42 GMT+0800 (China Standard Time)

We change the parameter type(String.class to byte[]) in ModifyRequestGatewayFactory and ModifyResponseGatewayFactory, then the error doesn't occurred. This might be bug in SpringCloudGateway?