Giters

sharkdp / hyperfine

A command-line benchmarking tool

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Bloated benchmark results

dzfrias opened this issue · comments

commented

I've been using hyperfine for a while now, and I've noticed some strange behavior on one of the projects that I've been measuring the performance for. I'm testing two binaries of almost the same program, but one is around 5 times the size of the other. There is no difference in behavior between the binaries. According to hyperfine, the smaller one executes around 20 times faster.

I'm aware that hyperfine should know nothing about the details of the two binaries, but a benchmark from bench tells me that they're executing in the same amount of time (which makes sense given that the binaries in question are functionally the exact same and have the exact same execution). I have used many different measuring methods, and they've all also led me to this conclusion. This has led me to believe that there is a problem in hyperfine.

Do you have any ideas as to why this could happen, or a place to pay particular attention to in the source code?

commented

Thank you for reporting this. I currently have no idea why this would happen. Is there any chance you can share those binaries? How do you run the benchmark? Which version of hyperfine do you use?

Have you tried benchmarking with --shell=none?

Why is one of the binaries bigger? Debugging information?

commented

I'll work on sharing the binaries when I strip away some of the noise of the other parts of the project. For the benchmark, the only argument I pass in is --warmup 5, with version 1.17.0 of hyperfine. I tried --shell=none and there's no big difference.

The bigger binary includes bindings to C++ code. When I don't include a few lines that use the bindings, the Rust compiler presumably optimizes the all the C++ stuff out.

I'll keep poking around hyperfine's source to get any clues, and I'll work on sharing the binaries when they're in a noisy form.

commented

Here are the binaries in a stripped-down form.

Since you might not want to download and run random executables (I personally definitely wouldn't), I can also work on a minimal reproduction of the behavior and share the source so you can build it locally. Building it locally shouldn't be too crazy, it's just a Rust project. The minimal reproduction might take me a bit longer, though, as the project is pretty big.

binaries.zip

commented

I just tested the binaries again, and it turns out that hyperfine isn't doing anything here. I'm terribly sorry for the inconvenience. In my initial tests, I must've not built the project properly. The problem seems to be on the project side, not with hyperfine. As to why it's happening in the project, I have no idea.

I'll close this issue

edit:
The only strange part is that bench hyperfine, and the Unix time command report the binaries running at the different speeds, while samply and Rust's builtin Time::instant report them having the same speed... do you have any ideas as to why this happens?

commented

I just tested the binaries again, and it turns out that hyperfine isn't doing anything here. I'm terribly sorry for the inconvenience.

No worries.

while samply and Rust's builtin Time::instant report them having the same speed

What does this mean? How exactly do you measure the full execution time with Time::instant? In the same process? If so, the startup time of the program is probably missing.