ByteBuf LEAK when PoolAcquirePendingLimitException is thrown
Akshay-Sundarraj opened this issue · comments
When executing a Http POST request if channel acquisition fails POST request body ByteBuf is not released and results in Netty LEAK
Expected Behavior
When error occurs reactor netty should release the buffer.
Actual Behavior
ByteBuf is not released and resulting in LEAK
Steps to Reproduce
I've uploaded sample program to reproduce the issue and also included logs which contain Netty LEAK detection.
https://gist.github.com/Akshay-Sundarraj/694647e5f32e72ccb6ee227c0c7d53b0
Your Environment
- Reactor version(s) used: 1.1.18
- JVM version (
java -version): 21.0.2 - OS and version (eg.
uname -a): 23.3.0 Darwin Kernel Version 23.3.0: Wed Dec 20 21:31:00 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T6020 arm64
@Akshay-Sundarraj You should always defer the creation of the buffers. Something like this:
static Mono<String> execute(HttpClient httpClient, String uri) {
return Mono.defer(() -> {
return Mono.defer(() -> {
return httpClient.request(HttpMethod.POST)
.uri(uri)
.send((httpClientRequest, nettyOutbound) -> {
return nettyOutbound.send(Mono.create(sink -> {
ByteBuf buffer = ByteBufAllocator.DEFAULT.directBuffer();
buffer.writeBytes("Hello World".getBytes(StandardCharsets.UTF_8));
sink.success(buffer);
}));
}).responseSingle(((response, byteBufMono) -> {
return byteBufMono.asString();
}));
}).doFinally(signalType -> {
});
});
}
@violetagg Thanks for the suggestion. The thing is I need to fill the buffer with some binary values based on some binary protocol somewhere else and filling the buffer(serialising some objects) may throw errors. Basically buffer is created and filled in some other part. Is there any way I can handle this?
@Akshay-Sundarraj I cannot help with creating the reactive pipeline as I do not know your project.
@violetagg My question is reactor netty release buffer in some cases and does not release is some other cases. How can I know in what scenarios it doesn't relases?
@Akshay-Sundarraj If we receive the buffer we will definitely release it, however here the HttpClient doesn't start sending at all as it cannot obtain a connection from the pool. That said the buffer consumption is lazy, we will not obtain the buffer as a first step. As a first step we need a connection that will be used for sending the request. Because of that you have to create the buffer only if there is a subscriber that will consume it, while now you create the buffer always regardless whether there is a subscriber or not.
@violetagg Thanks for the reply. I'll try to update the code as suggested by you.
@violetagg I followed your suggestion of creating ByteBuf only when channel is acquired. i.e. inside send(). But now I'm getting different LEAK. Now I added TImeout to Http response Mono and resulting in ByteBuf leaks.
Minimal steps to reproduce the issue is available here https://gist.github.com/Akshay-Sundarraj/5bae74c268835ded6f128016c525cc2b
echoclient.log
I'm also uploading the collected logs to this message.
@Akshay-Sundarraj The client should look like this:
static Mono<String> execute(HttpClient httpClient, String uri) {
return httpClient.post().uri(uri).send(((httpClientRequest,
nettyOutbound) -> {
int request = requestId.getAndIncrement();
httpClient.headers(headers -> headers.set("id", 1));
return nettyOutbound.send(
Mono.fromCallable(() -> {
ByteBuf buf = nettyOutbound.alloc().directBuffer()
.writeBytes(("Hello World " + Integer.toString(request)) .getBytes(StandardCharsets.UTF_8));
logger.fine("ByteBuffer refCount for request " + request +
" is " + buf.refCnt());
return buf;
})
.doOnNext(buf1 -> logger.fine("buffer is emitted for request " + request))
).then();
})).responseSingle(((response, byteBufMono) -> byteBufMono.asString())).
doFinally(signalType -> logger.fine("Http Mono is " +
"completed with " + signalType));
}
@violetagg Thanks. This fixed the issue.