LEAK: ByteBuf.release() was not called before it's garbage-collected.

Question

LEAK: ByteBuf.release() was not called before it's garbage-collected.

subham611 opened this issue 6 months ago · comments

Expected Behavior

I am using spring boot 2.6.13 version with spring-boot-starter-webflux dependency.
During low request count spring server works fine but when rps increased I am facing issue related to BUFFER LEAK.

Actual Behavior

Getting below mentioned error:
LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information. Recent access records: Created at:io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:403) io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188) io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:116) org.springframework.core.io.buffer.NettyDataBufferFactory.allocateBuffer(NettyDataBufferFactory.java:71) org.springframework.core.io.buffer.NettyDataBufferFactory.allocateBuffer(NettyDataBufferFactory.java:39) org.springframework.core.codec.CharSequenceEncoder.encodeValue(CharSequenceEncoder.java:91) org.springframework.core.codec.CharSequenceEncoder.lambda$encode$0(CharSequenceEncoder.java:75) reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:113) reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2398) reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171) reactor.core.publisher.MonoSingle$SingleSubscriber.doOnRequest(MonoSingle.java:103) reactor.core.publisher.Operators$MonoInnerProducerBase.request(Operators.java:2731) reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2194) reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2068) reactor.core.publisher.MonoSingle$SingleSubscriber.onSubscribe(MonoSingle.java:115) reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96) reactor.core.publisher.FluxJust.subscribe(FluxJust.java:68) reactor.core.publisher.Mono.subscribe(Mono.java:4455) reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:200) reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53) reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57) reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157) reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816) reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151) reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816) reactor.core.publisher.MonoZip$ZipCoordinator.signal(MonoZip.java:251) reactor.core.publisher.MonoZip$ZipInner.onNext(MonoZip.java:336) reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2398) reactor.core.publisher.MonoZip$ZipInner.onSubscribe(MonoZip.java:325) reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55) reactor.core.publisher.Mono.subscribe(Mono.java:4455) reactor.core.publisher.MonoZip.subscribe(MonoZip.java:128) reactor.core.publisher.Mono.subscribe(Mono.java:4455) reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103) reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:544) reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121) reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192) reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259) reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121) reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258) reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192) reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259) reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onError(FluxPeekFuseable.java:234) reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258) reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onError(MonoIgnoreElements.java:84) reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onError(FluxConcatArray.java:207) reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121) reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:142) reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129) reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107) reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.onNext(FluxPeekFuseable.java:503) reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816) reactor.core.publisher.MonoCallable.subscribe(MonoCallable.java:62) reactor.core.publisher.Mono.subscribe(Mono.java:4455) reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onComplete(FluxConcatArray.java:258) reactor.core.publisher.FluxConcatArray.subscribe(FluxConcatArray.java:78) reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157) reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107) reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2398) reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.request(FluxContextWrite.java:136) reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:110) reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onSubscribe(FluxContextWrite.java:101) reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55) reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157) reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74) reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2664) reactor.core.publisher.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:180) reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:152) reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2400) reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171) reactor.core.publisher.MonoSingle$SingleSubscriber.doOnRequest(MonoSingle.java:103) reactor.core.publisher.Operators$MonoInnerProducerBase.request(Operators.java:2731) reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2194) reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2068) reactor.core.publisher.MonoSingle$SingleSubscriber.onSubscribe(MonoSingle.java:115) reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96) reactor.core.publisher.FluxJust.subscribe(FluxJust.java:68) reactor.core.publisher.Mono.subscribe(Mono.java:4455) reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:200) reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53) reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57) reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157) reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816) reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:249) reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79) reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:200) reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:200) reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.complete(MonoIgnoreThen.java:292) reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onNext(MonoIgnoreThen.java:187) reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816) reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151) reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816) reactor.core.publisher.MonoZip$ZipCoordinator.signal(MonoZip.java:251) reactor.core.publisher.MonoZip$ZipInner.onNext(MonoZip.java:336) reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180) reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:101) reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74) reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79) reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816) reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151) reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107) reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299) reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337) reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816) reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:160) reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144) reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260) reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144) reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:400) reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:419) reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:603) reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:266) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:336) io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:308) io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

JVM version (java -version): 17
spring boot version: 2.6.13

Violeta Georgieva · Answer 1 · Wed Apr 03 2024 14:58:45 GMT+0800 (China Standard Time)

@subham611 Please follow this FAQ and provide the extended logs
https://projectreactor.io/docs/netty/release/reference/index.html#faq.memory-leaks

Violeta Georgieva · Answer 2 · Wed Apr 03 2024 15:00:41 GMT+0800 (China Standard Time)

@subham611 Consider checking Spring Boot supported versions
https://spring.io/projects/spring-boot#support

github-actions · Answer 3 · Thu Apr 11 2024 14:26:25 GMT+0800 (China Standard Time)

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Subham Singhal · Answer 4 · Mon Apr 15 2024 17:47:58 GMT+0800 (China Standard Time)

I updated spring boot to latest version 3.2.4 still getting this error.
I have added following jvm parameters, and not seeing any additional log

-XX:MaxDirectMemorySize=1024M
-Dio.netty.leakDetection.targetRecords=40
-Dio.netty.leakDetection.level=paranoid
-Dlogging.level._reactor.netty.channel.LeakDetection=debug

Do I need to set logging.level._reactor.netty.channel.LeakDetection=debug in application.properties.

Violeta Georgieva · Answer 5 · Mon Apr 15 2024 17:59:09 GMT+0800 (China Standard Time)

@subham611 Try to set it in application.properties

Subham Singhal · Answer 6 · Mon Apr 15 2024 18:00:53 GMT+0800 (China Standard Time)

yeah, saw this #1874 (comment)
Setting following properties in application.properties file

spring.netty.leak-detection=paranoid
logging.level._reactor.netty.channel.LeakDetection=DEBUG

Subham Singhal · Answer 7 · Mon Apr 15 2024 19:47:44 GMT+0800 (China Standard Time)

@violetagg I set above mentioned spring properties but not seeing any change in log message

{"timestamp":1713180562148,"thread":"reactor-http-epoll-1","logger":"io.netty.util.ResourceLeakDetector","message":"LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:404)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
	io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:116)
	org.springframework.core.io.buffer.NettyDataBufferFactory.allocateBuffer(NettyDataBufferFactory.java:72)
	org.springframework.core.io.buffer.NettyDataBufferFactory.allocateBuffer(NettyDataBufferFactory.java:39)
	org.springframework.core.codec.CharSequenceEncoder.encodeValue(CharSequenceEncoder.java:90)
	org.springframework.core.codec.CharSequenceEncoder.lambda$encode$0(CharSequenceEncoder.java:74)
	reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:113)
	reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571)
	reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)
	reactor.core.publisher.MonoSingle$SingleSubscriber.doOnRequest(MonoSingle.java:103)
	reactor.core.publisher.Operators$MonoInnerProducerBase.request(Operators.java:2909)
	reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2367)
	reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2241)
	reactor.core.publisher.MonoSingle$SingleSubscriber.onSubscribe(MonoSingle.java:115)
	reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)
	reactor.core.publisher.FluxJust.subscribe(FluxJust.java:68)
	reactor.core.publisher.Mono.subscribe(Mono.java:4568)
	reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:202)
	reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53)
	reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:63)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165)
	reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158)
	reactor.core.publisher.MonoZip$ZipCoordinator.signal(MonoZip.java:297)
	reactor.core.publisher.MonoZip$ZipInner.onNext(MonoZip.java:478)
	reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571)
	reactor.core.publisher.MonoZip$ZipInner.onSubscribe(MonoZip.java:470)
	reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55)
	reactor.core.publisher.MonoZip$ZipCoordinator.request(MonoZip.java:220)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
	reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2367)
	reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onSubscribe(FluxOnErrorResume.java:74)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
	reactor.core.publisher.MonoZip.subscribe(MonoZip.java:129)
	reactor.core.publisher.Mono.subscribe(Mono.java:4568)
	reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103)
	reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:544)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:241)
	reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:315)
	reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:241)
	reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:315)
	reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121)
	reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:241)
	reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:315)
	reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onError(FluxPeekFuseable.java:234)
	reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258)
	reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onError(MonoIgnoreElements.java:84)
	reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onError(FluxConcatArray.java:208)
	reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121)
	reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:241)
	reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:315)
	reactor.netty.FutureMono$FutureSubscription.operationComplete(FutureMono.java:193)
	io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
	io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)
	io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)
	io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
	io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
	io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629)
	io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118)
	io.netty.util.concurrent.PromiseCombiner.tryPromise(PromiseCombiner.java:170)
	io.netty.util.concurrent.PromiseCombiner.access$600(PromiseCombiner.java:35)
	io.netty.util.concurrent.PromiseCombiner$1.operationComplete0(PromiseCombiner.java:62)
	io.netty.util.concurrent.PromiseCombiner$1.operationComplete(PromiseCombiner.java:44)
	io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
	io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557)
	io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
	io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
	io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629)
	io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118)
	io.netty.util.internal.PromiseNotificationUtil.tryFailure(PromiseNotificationUtil.java:64)
	io.netty.channel.ChannelOutboundBuffer.safeFail(ChannelOutboundBuffer.java:754)
	io.netty.channel.ChannelOutboundBuffer.remove0(ChannelOutboundBuffer.java:339)
	io.netty.channel.ChannelOutboundBuffer.failFlushed(ChannelOutboundBuffer.java:691)
	io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:735)
	io.netty.channel.AbstractChannel$AbstractUnsafe.handleWriteError(AbstractChannel.java:950)
	io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:933)
	io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:570)
	io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
	io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)
	io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

Violeta Georgieva · Answer 8 · Mon Apr 15 2024 20:01:05 GMT+0800 (China Standard Time)

@subham611 It doesn't seem that the changes are applied.

Violeta Georgieva · Answer 9 · Mon Apr 15 2024 20:02:13 GMT+0800 (China Standard Time)

You should be able to see output similar to the one here #1874 (comment)

Subham Singhal · Answer 10 · Mon Apr 15 2024 22:07:54 GMT+0800 (China Standard Time)

Is there any other way to set netty property?

Violeta Georgieva · Answer 11 · Mon Apr 15 2024 22:16:48 GMT+0800 (China Standard Time)

@subham611 Please consult with the reference documentation of the technologies that you use. For Spring Boot - it should be application.properties (there might be different versions of this - for testing, production etc), then the environment where you run your application - whether the provided JVM OPTS are applied etc.

I cannot help without knowing the environment where you run your app.

You should be able to check the exact JVM OPTS that are used to run your app and the binaries that are published.

Subham Singhal · Answer 12 · Tue Apr 16 2024 01:15:45 GMT+0800 (China Standard Time)

I have added this flag -Dlogging.level.io.netty.util.ResourceLeakDetector=debug as well and i do see following logs

{"timestamp":1713191024585,"thread":"main","logger":"io.netty.util.ResourceLeakDetector","message":"-Dio.netty.leakDetection.level: paranoid","context":"default","log_level":"DEBUG","is_logging_enabled":false}
{"timestamp":1713191024585,"thread":"main","logger":"io.netty.util.ResourceLeakDetector","message":"-Dio.netty.leakDetection.targetRecords: 40","context":"default","log_level":"DEBUG","is_logging_enabled":false}

Which confirms that leakDetection level is set to paranoid but somehow I am not seeing full logs during error.

I have setup a rate limiter using webFilter and I am suspecting some issue there.

@Component
public class RateLimiterFilter implements WebFilter {
  private final RateLimiter rateLimiter;
  @Override
    public Mono<Void> filter(ServerWebExchange exchange, WebFilterChain chain) {
       if(!rateLimiter.acquirePermission()) {
           log.warn("Too many requests");
           exchange.getResponse().setStatusCode(HttpStatus.TOO_MANY_REQUESTS);
           return Mono.empty(); // I am suspecting this might be causing some issue. 
       }
       return chain.filter(exchange);
    }
}

Let me remove this and try. ref: #518 (comment)

github-actions · Answer 13 · Tue Apr 23 2024 14:27:45 GMT+0800 (China Standard Time)

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Subham Singhal · Answer 14 · Tue Apr 23 2024 14:40:44 GMT+0800 (China Standard Time)

Above change didn't help.
I tried setting ResourceLeakDetector property programmatically, that didn't help either

 @Bean
    public void nettyServerCustomizer(){
        ResourceLeakDetector.setLevel(ResourceLeakDetector.Level.PARANOID);
        if(ResourceLeakDetector.isEnabled()) {
            System.out.println(ResourceLeakDetector.getLevel());
        }
    }

Violeta Georgieva · Answer 15 · Tue Apr 23 2024 15:00:32 GMT+0800 (China Standard Time)

@subham611 Please provide a reproducible example so that we can investigate further.

github-actions · Answer 16 · Thu May 02 2024 14:26:46 GMT+0800 (China Standard Time)

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

github-actions · Answer 17 · Thu May 09 2024 14:27:24 GMT+0800 (China Standard Time)

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open.