Giters

hyperium / h2

HTTP 2.0 client & server implementation for Rust.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Make tracing an optional feature

seanmonstar opened this issue · comments

commented

Make tracing available as a Cargo feature, probably default off.

commented

While test integrating tonic into TiKV to replace grpc-rs, I found that tonic's performance is not as good as grpc-rs as it consumes more cpu. And from the profile, I saw that a lot of cpu time is cost by the tracing logic in h2. In some workloads, the qps with tonic can be 15% lower than grpc-rs. Here is a cpu profiling snapshot:

image

After patching tonic with a h2 package with all tracing code removed, tonic's performance is slightly better than gprc-rs in almost all workload without any further optimization.

commented

There's something about this that surprises me. Namely, should tracing really incur a 15% overhead if the tracing level is low? That seems shocking. @glorv any interest in providing repro steps or raw profiles?

commented

There's something about this that surprises me. Namely, should tracing really incur a 15% overhead if the tracing level is low? That seems shocking. @glorv any interest in providing repro steps or raw profiles?

@ajwerner It's an attempt to replace grpc-rs with tonic for tikv. The workload is sysbench 's oltp_write_only that is a CPU heavy workload with high write concurrency(256 in my benchmark). So any part consumes more CPU will result with a performance regression.

Here are 2 profiles with the same workload, the 2nd is patched with all tracing code removed.

profile.zip
(Because github doesn't allow to paste raw .proto file, I put the two profile samples in this zip).

commented

Thanks, that's helpful context, and sad. It feels to me like the subscriber should have indicated that the span is not enabled and the span never should have been created. I wonder how you're setting up your tracing subscriber. I wonder if you set it up such that all of the tracing metadata from this crate are not marked enabled when returning from https://docs.rs/tracing/latest/tracing/trait.Subscriber.html#tymethod.enabled, whether you'd see any of this overhead.