Memory leak caused by the long-lifespan trace_debug_span within the reused connection
ikhin opened this issue · comments
We reuse the HTTP2 Connection created by hyperium/h2.
However, During the stress testing, an issue of memory leakage occurred, causing a crash and reboot of the system. So I begin the troubleshooting.
![截屏2024-04-11 20 23 20](https://private-user-images.githubusercontent.com/20532424/321656518-8e5f0349-b026-4bff-b979-8ae3ed49cd64.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk5MDQ2MzAsIm5iZiI6MTcxOTkwNDMzMCwicGF0aCI6Ii8yMDUzMjQyNC8zMjE2NTY1MTgtOGU1ZjAzNDktYjAyNi00YmZmLWI5NzktOGFlM2VkNDljZDY0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MDIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzAyVDA3MTIxMFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTkwZGI0ZmE5Y2JiYjcxZTBhMDlhYzFkY2M4OTFkMDZmOGI5NWZlM2ZiNTk5YmQ3MjllMmEwMDQ4ZWUzYmEwOWImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.8Hz1o_1FTfINxltaoWaNU7P8KD1kcXW8gNlxnivMjiA)
1. Start Troubleshooting with the memory growth chart
First, using Jemalloc
and Jeprof
, I generated memory growth chart at intervals. jeprof_memory_leak.pdf
The largest memory block was finish_grow
, which occupied 88.8% of the total allocation on its own. The finish_grow
function is related to Vec reallocating new memory when expanding. This may indicate that the memory leak occurred during the process of writing to Vec, where memory was allocated to these Vecs but was not released in a timely manner.
I found the function call chain upstream of finish_grow
is tracing_core -> event -> Event -> dispatch
, which is related to the subscriber
. In our business code, the subscriber
specified for tracing_core to use is opentelemetry
.
let tracer = opentelemetry_jaeger::new_collector_pipeline()
.with_endpoint(url)
.with_service_name(server_name)
.with_hyper()
.install_batch(opentelemetry::runtime::Tokio)?;
tracing_subscriber::registry()
.with(tracing_opentelemetry::layer().with_tracer(tracer))
.try_init()?;
In opentelemetry_api
, I found the function call chain finish_grow <- reserve_for_push <- push
at function on_event
, which includes operations for pushing events onto the events
array in a span. tracing_opentelemetry/layer.rs
fn on_event(&self, event: &Event<'_>, ctx: Context<'_, S>) {
if let Some(span) = ctx.lookup_current() {
...
if let Some(ref mut events) = builder.events {
events.push(otel_event);
} else {
builder.events = Some(vec![otel_event]);
}
...
}
2. Start Troubleshooting with Jaeger on the other side
On the other hand, starting from Jaeger
, I observed a span that is as long as the pressure test duration. This span is a trace_debug_span
that was created when establishing a Connection.h2/src/proto/connection.rs
I've observed that there are many events
under the trace_debug_span
, as if an event
is added for every request that comes in.
h2/src/codec/framed_read.rs
h2/src/codec/framed_write.rs
To disable the trace_debug_span
in the h2
library, you can change the RUST_LOG
logging level from debug
to info
. After doing this and running a stress test again, you can observe that the memory no longer leaks.
In summary
I used the h2
library to create a persistent Connection
for reuse, which results in the creation of a trace_debug_span
that lasts as long as the lifespan of the connection. When a stress test is conducted with continuous requests, events are constantly added to the events
array inside the trace_debug_span
. As long as the persistent Connection
is not closed, the trace_debug_span
will not terminate and be destroyed after reporting. Therefore, the memory used by the trace_debug_span
keeps increasing, leading to a memory leak.
Although we normally don't enable RUST_LOG: "debug"
in production, sometimes it's necessary to turn it on for troubleshooting business issues based on debug logs. This memory leak issue has been troublesome.
Therefore, when creating a connection by default, is it possible to allow users to set trace_debug_span
as an optional feature?
I don't think this is h2's fault, it rather the subscriber which is growing the events forever. Any subscriber implementation will eventually need to cap its own growth.