First bench is always slower. Need warmup?

[◬]

The first bench call always seems to be slower than other tests. I've included a simple test case below.

benchmark.js:

require('matcha');
[1, 2, 3, 4].forEach((num) => {
  suite(`Benchmark #${num}`, () => {
    bench('empty', () => {});
    bench('empty', () => {});
  });
});

command line:

npm install -g matcha
matcha ./benchmark.js

output:

                      Benchmark #1
     414,441,918 op/s » empty
     126,934,245 op/s » empty

                      Benchmark #2
     130,458,368 op/s » empty
     132,597,178 op/s » empty

                      Benchmark #3
     132,507,794 op/s » empty
     131,705,064 op/s » empty

                      Benchmark #4
     135,100,324 op/s » empty
     131,982,266 op/s » empty


  Suites:  4
  Benches: 8
  Elapsed: 10,695.31 ms

I would expect all the benchmarks to execute around the same amount of time.

Thanks for this library btw!

[Evan Carroll]

This framework is no good. I suggest people avoid it.

https://www.reddit.com/r/node/comments/54rueh/using_let_significantly_slower_vs_var_v8_disables/
https://gist.github.com/Salakar/6d7b84f7adf1f3bc62a754752a6e5d0e

[Steven Langbroek]

@EvanCarroll rather than bashing the project without any justification other than this bug (I followed your link, where you also don't give any other explanation), you could also help figure out what's causing this behavior.

[Connor Peet]

As this issue (and other issues and PRs) have been open for years without update, I've created a newer rewrite of this tool. It uses the widely-used Benchmark.js, which includes warmup.