Giters

reactor / reactor-netty

TCP/HTTP/UDP/QUIC client/server with Reactor over Netty

Home Page:https://projectreactor.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

LEAK: ByteBuf.release() was not called before it's garbage-collected.

cobar79 opened this issue · comments

commented 
    public Optional<Path> downloadAndUnZip(String url) throws MyClientException, MyFileException {
        String baseUrl =url;
        if (baseUrl == null) {
            throw new MyClientException("Invalid URL");
        }
        String uri = url.substring(url.indexOf(baseUrl) + baseUrl.length());
        log.info("Download uri: {}", uri);
        Mono<DataBuffer> dataBufferFlux = fetchDownloadAndUnZip(uri);
        dataBufferFlux.block();
        String exportZipName = uri.substring(uri.lastIndexOf("/"));
        return fileService.writeZipFile(dataBufferFlux, exportZipName);
    }

    private Mono<DataBuffer> fetchDownloadAndUnZip(String uri) {
        return myWebClient.get()
            .uri(uriBuilder -> uriBuilder
                .path(Objects.requireNonNull(uri))
                .build()
            )
            .accept(MediaType.APPLICATION_OCTET_STREAM)
            .retrieve()
            .onStatus(status -> status.value() == HttpStatus.NOT_FOUND.value(), clientResponse -> {
                log.error(uri + " " + HttpStatus.NOT_FOUND + " Error");
                return Mono.error(new MyClientException(HttpStatus.NOT_FOUND.value() + ":: " + uri + " NOT FOUND"));
            })
            .bodyToMono(DataBuffer.class)
            .retryWhen(Retry.backoff(myConfig.getDownloadRetryMax(), Duration.ofSeconds(myConfig.getDownloadWaitMillis()))
                .filter(MyClientException.class::isInstance)
                .onRetryExhaustedThrow((retryBackoffSpec, retrySignal) -> {
                    String message = HttpStatus.NOT_FOUND.value() + ":: " + uri + " NOT FOUND. Failed to download after " + myConfig.getDownloadRetryMax() + " retries";
                    throw new MyClientException(message);
                })
            )
            .doFinally(signalType -> {
                log.warn("Operation completed with signal type: " + signalType);

                // Cleanup logic or additional actions can be performed here

                // For example, if you need to log or perform some action based on success or error:
                if (signalType == SignalType.ON_COMPLETE) {
                    log.info("Operation completed successfully");
                } else if (signalType == SignalType.ON_ERROR) {
                    log.error("Operation completed with an error");
                }
            });
    }

Writing the Zip file:

    public Optional<Path> writeZipFile(Mono<DataBuffer> dataBuffer, String file) throws MyFileException {
        Path tempFilePath = Paths.get(getDownloadPath().toAbsolutePath() + File.separator + file);
        DataBufferUtils.write(dataBuffer, tempFilePath, StandardOpenOption.CREATE).block();
        log.debug("Zip file @ {}", tempFilePath.toAbsolutePath());
        return unzipUpdates(tempFilePath);
    }

Logs:

2024-15-01 12:16:01.060 [scheduling-1] INFO  com.dp.ingest.client.MyClientServiceImpl.downloadAndUnZip - Download uri: /20240115191500.export.CSV.zip

2024-15-01 12:16:01.505 [reactor-http-nio-2] INFO  com.dp.ingest.config.rest.MyWebClientConfig.lambda$logResponse$6 - Response: 200 OK

2024-15-01 12:16:01.507 [reactor-http-nio-2] WARN  com.dp.ingest.client.MyClientServiceImpl.lambda$fetchDownloadAndUnZip$7 - Operation completed with signal type: onComplete

2024-15-01 12:16:01.518 [reactor-http-nio-2] INFO  com.dp.ingest.client.MyClientServiceImpl.lambda$fetchDownloadAndUnZip$7 - Operation completed successfully

2024-15-01 12:16:01.556 [scheduling-1] DEBUG com.dp.ingest.service.impl.MyFileServiceImpl.writeZipFile - Zip file @ C:\projects\os\ingest\app\.\temp\20240115191500.export.CSV.zip

2024-15-01 12:16:01.567 [scheduling-1] INFO  com.dp.ingest.service.impl.MyFileServiceImpl.unzipUpdates - Entry 20240115191500.export.CSV

2024-15-01 12:16:01.568 [scheduling-1] DEBUG com.dp.ingest.service.impl.MyFileServiceImpl.unzipUpdates - ENTRY AT .\temp\20240115191500.export.CSV

2024-15-01 12:16:01.572 [scheduling-1] INFO  com.dp.ingest.service.impl.MyIngestServiceImpl.processExportUpdates - Parsing .\temp\20240115191500.export.CSV, length = 524691

2024-15-01 12:16:02.129 [scheduling-1] INFO  com.dp.ingest.service.impl.PublishServiceImpl.queueEvents - Queueing 1302 events.

2024-15-01 12:18:47.949 [scheduling-1] INFO  com.dp.ingest.service.impl.PublishServiceImpl.queueEvents - Queued 1302 events to parse-events

2024-15-01 12:18:47.951 [scheduling-1] INFO  com.dp.ingest.service.impl.MyIngestServiceImpl.processExportUpdates - *** Processed 1302 events in 166891 millis.

2024-15-01 12:18:47.955 [scheduling-1] INFO  com.dp.ingest.service.impl.MyFileServiceImpl.deleteTsvFile - DELETED:: .\temp\20240115191500.export.CSV

2024-15-01 12:18:47.956 [scheduling-1] DEBUG com.dp.ingest.service.impl.MyFileServiceImpl.deleteZipFile - delete url /20240115191500.export.CSV.zip

2024-15-01 12:18:47.960 [scheduling-1] INFO  com.dp.ingest.service.impl.MyFileServiceImpl.deleteZipFile - DELETED:: /20240115191500.export.CSV.zip

2024-15-01 12:18:47.963 [scheduling-1] INFO  com.dp.ingest.service.impl.MyJob.executeJob - Finished Job in 167954 millis

2024-15-01 12:20:01.485 [reactor-http-nio-2] ERROR io.netty.util.ResourceLeakDetector.reportTracedLeak - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
#1:
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:300)
        io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:833)
#2:
        io.netty.buffer.AdvancedLeakAwareByteBuf.getByte(AdvancedLeakAwareByteBuf.java:155)
        io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:1063)
        io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:656)
        io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:285)
        io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:239)
        io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:833)
#3:
        io.netty.buffer.AdvancedLeakAwareByteBuf.indexOf(AdvancedLeakAwareByteBuf.java:647)
        io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:1051)
        io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:656)
        io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:285)
        io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:239)
        io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:833)
#4:
        io.netty.buffer.AdvancedLeakAwareByteBuf.getByte(AdvancedLeakAwareByteBuf.java:155)
        io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:1063)
        io.netty.handler.codec.http.HttpObjectDecoder$LineParser.parse(HttpObjectDecoder.java:1113)
        io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:270)
        io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:239)
        io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:833)
#5:
        io.netty.buffer.AdvancedLeakAwareByteBuf.indexOf(AdvancedLeakAwareByteBuf.java:647)
        io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:1051)
        io.netty.handler.codec.http.HttpObjectDecoder$LineParser.parse(HttpObjectDecoder.java:1113)
        io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:270)
        io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:239)
        io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
        io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:833)
#6:
        Hint: 'reactor.left.httpCodec' will handle the message from this point.
        io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:417)
        io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:833)
#7:
        Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
        io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
        io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:417)
        io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:833)
#8:
        io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:635)
        io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:833)
Created at:
        io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:403)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
        io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:140)
        io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:150)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        java.base/java.lang.Thread.run(Thread.java:833)
: 1 leak records were discarded because they were duplicates
: 41 leak records were discarded because the leak record count is targeted to 4. Use system property io.netty.leakDetection.targetRecords to increase the limit.

Note the signal type logs from the WebClient finally block.

Expected Behavior

More indication where the exception occurred.

Actual Behavior

Dumps a bunch of logs with no indication where in the application code the ByteBuf was created.

Steps to Reproduce

downloadAnUnzip("http://localhost/export.CSV.zip")

Possible Solution

Your Environment

Running with Spring Boot v3.1.4, Spring v6.0.12

  • Reactor version(s) used: 1.1.11
  • Other relevant libraries versions (eg. netty, ...): 4.1.97
  • JVM version (java -version): 17
  • OS and version (eg. uname -a): MINGW64_NT-10.0-19044
commented

Investigating if the exception is coming from AWS SQS client 2.20.63

commented

@cobar79 How to debug memory leaks please follow this FAQ

commented

@cobar79 This code is interesting

public Optional<Path> downloadAndUnZip(String url) throws MyClientException, MyFileException {
        String baseUrl =url;
        if (baseUrl == null) {
            throw new MyClientException("Invalid URL");
        }
        String uri = url.substring(url.indexOf(baseUrl) + baseUrl.length());
        log.info("Download uri: {}", uri);
        Mono<DataBuffer> dataBufferFlux = fetchDownloadAndUnZip(uri);
-->  dataBufferFlux.block();
        String exportZipName = uri.substring(uri.lastIndexOf("/"));
        return fileService.writeZipFile(dataBufferFlux, exportZipName);
    }

First you block which will retrieve the DataBuffer and then you again try to process dataBufferFlux, I might be missing something here?

commented

@violetagg Thank you for the observation. I missed the block on the DataBufferUtils.write.