Use BenchmarkDotNet
george-polevoy opened this issue · comments
My application requires that performance of a hash function invocation is measured in fractions of a nanosecond, because I plan to use hash computation in compiler development. This code can potentially used in runtime when the actual hash implementation is generated, and actual invocation runs under 2.5 nanoseconds on a modern Intel CPU.
A good benchmarking harness should do every possible effort to factor out any slowdown.
Such as slowdown can arise from JIT compilation, Tiered JIT compilation, invocation overhead, conditional logic in loops, etc.
I noticed that a very basic custom code is used for benchmarking harness.
This way it is very difficult to get a precise result especially for short payloads such as an average string length used in identifiers.
There are many problems with this benchmarking code that affects precision of benchmarks.
Current implementation uses delegates for invocations. Delegates themselves can use tens of CPU cycles.
Current implementation does not use loop unrolling, so measurement is greatly affected by the loop conditional logic.
Current implementation does not use factoring out the test harness runtime from the results.
I believe that is the reason why this benchmark only tests throughput, not performance of invocations on small payloads, such as small or bytes arrays.
Another issue with custom code that it is really hard to get results for various platforms, frameworks and SDKs.
BenchmarkDotNet is a de facto industry standard. NET Core teams use BDN for their own benchmarks.
It provides loop unrolling, preflight runs, special runs to compute actual overhead in function calling, statistical measurements etc.
Also it provides a convenient API, cross-platform, cross-SDK and reporting capabilities.
Thanks @george-polevoy, yes I've tried BenchmarkDotNet before, but it takes too long to come up with results. I agree that these benchmarks aren't as accurate though. I'm relying on benchmarks for certain iterative optimization work so a quick check helps me a lot. Maybe we can have two benchmark suites in the solution for different purposes.
We can come up with command line parameters to run faster.
One problem with short runs is that it's not just measured imprecisely, but can be totally misleading because of Tiered JIT, which might kick in on a later stage. This can only happen if a method is invoked multiple times. This actually happened to my own benchmarks.
--job Short or --job Dry should do the job for quick iterations.
Fixed as of 3.1.0.