Giters

sharkdp / hyperfine

A command-line benchmarking tool

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Support --time-unit microsecond

titzer opened this issue · comments

commented

Apologies if this may have been discussed before, but it would be very useful to have hyperfine report microsecond resolution for times, as this is generally the time unit reported by the underlying system calls, e.g. wait4 on POSIX.

Elsewhere, IIRC, it was stated that execution times exhibit too much variance for this to be useful. I'm not sure this is true; e.g., frustrated by the second and millisecond resolution of the time command on most UNIXes, I did experiments and wrote a utility called btime that reports microseconds. I found that for very short-running programs with a significant number of repeated runs, I was able to reliably characterize statistical distributions down into the hundreds or even tens of microseconds range. Regardless of that distribution, from a tool perspective, I think it makes sense for hyperfine to report measurements at the resolution of the underlying timers rather than round measurements without recourse.

Thanks for considering.

commented

Elsewhere, IIRC, it was stated that execution times exhibit too much variance for this to be useful

This sounds like something I might have written in the past, but it's probably not true anymore. Previously, hyperfine always needed to launch a new shell for each benchmark execution. We do correct times and subtract that shell spawning overhead, but my feeling was that this procedure added noise in the order of 100µs — 1ms, which is why I recommended not to benchmark programs running faster than 5 ms — just to be cautious with my claims on what this program can do.

With the new --shell=none/-N option of hyperfine, that restriction is now gone. We do not need intermediate shells to run the benchmarkee. So I can certainly imagine that your observation …

I found that for very short-running programs with a significant number of repeated runs, I was able to reliably characterize statistical distributions down into the hundreds or even tens of microseconds range

… is correct, and I would definitely agree that we should add microseconds as a new option for the --time-unit argument.

Thank you for your feedback.

commented

Awesome, thanks!