DefaultHttp2HeadersEncoder corrupts unpooled memory metrics?

[Stanislav Kashirin]

Expected behavior

Heap memory counter in UnpooledByteBufAllocatorMetric correlates with the actual heap consumption.

Actual behavior

Heap memory counter is always growing, but there is no evidence of the genuine memory leak.

Steps to reproduce

Minimal yet complete reproducer code (or URL to code)

I may craft an executable example if it is needed, but I believe this is the already known issue (sort of).

Netty version

4.1.63.Final, but I guess it's still here

JVM version (e.g. java -version)

openjdk 17.0.3 (custom build, but it is irrelevant here)

OS version (e.g. uname -a)

Linux <redacted> 5.4.161-26.3 #1 SMP Mon Feb 7 14:47:58 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Hi!

I'm using gRPC (and as a result Netty HTTP2 implementation as well).
Also, I'm tracking the memory consumption of pooled/unpooled heap/offheap netty allocators by using built-in allocator metrics.
And right now, I observe some peculiar behaviour: unpooled heap counter is always growing like the memory is leaking, but the actual heap consumption of JVM does not correllate with this metric.

After some debugging I've found the only place there unpooled allocator is used in my app:

Breakpoint reached
	at io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeHeapByteBuf.allocateArray(UnpooledByteBufAllocator.java:145)
	at io.netty.buffer.UnpooledHeapByteBuf.<init>(UnpooledHeapByteBuf.java:59)
	at io.netty.buffer.UnpooledUnsafeHeapByteBuf.<init>(UnpooledUnsafeHeapByteBuf.java:34)
	at io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeHeapByteBuf.<init>(UnpooledByteBufAllocator.java:139)
	at io.netty.buffer.UnpooledByteBufAllocator.newHeapBuffer(UnpooledByteBufAllocator.java:82)
	at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:168)
	at io.netty.buffer.AbstractByteBufAllocator.heapBuffer(AbstractByteBufAllocator.java:154)
	at io.netty.buffer.Unpooled.buffer(Unpooled.java:101)
	at io.netty.handler.codec.http2.DefaultHttp2HeadersEncoder.<init>(DefaultHttp2HeadersEncoder.java:30)
	at io.netty.handler.codec.http2.DefaultHttp2HeadersEncoder.<init>(DefaultHttp2HeadersEncoder.java:37)
	at io.netty.handler.codec.http2.DefaultHttp2HeadersEncoder.<init>(DefaultHttp2HeadersEncoder.java:33)
	at io.netty.handler.codec.http2.DefaultHttp2FrameWriter.<init>(DefaultHttp2FrameWriter.java:88)
	at io.grpc.netty.NettyServerHandler.newHandler(NettyServerHandler.java:169)
	at io.grpc.netty.NettyServerTransport.createHandler(NettyServerTransport.java:263)
	at io.grpc.netty.NettyServerTransport.start(NettyServerTransport.java:131)
	at io.grpc.netty.NettyServer$1.initChannel(NettyServer.java:290)
	at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129)
	at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112)
	at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:938)
	at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609)
	at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46)
	at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463)
	at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115)
	at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:514)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429)
	at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:833)

Here I've spotted that the similar problem with the memory leaks is already known in HTTP2 module for v5: #12781
I guess that we have the same problem in 4.x version too.

The tableSizeChangeOutput is allocated as unpooled buf there, so it does not lead to the actual memory leak, only corrupts the mem counter (always incremented but never decremented).

I'd like to make a PR and backport the fix if you are ok with that.