Eagerly abort hyperfine run when output differs

[Gunnar Morling]

Hey @hundredwatt, not sure whether it's doable: but could we abort the run for a given contender when its output differs in the first run (or warmup even)?

[Alexander Yastrebov]

If we use measurements_1B.out instead of out_expected.txt (make out_expected.txt a link to it if really needed) - then it follows input/output convention supported by test.sh so we can do ./test <fork> measurements_1B.txt before hyperfine (and instead of warmup).

[Jason Nochlin]

I think easiest is to perform the warmup run w/o hyperfine, check the output, then use hyperfine for the 5 runs

I keep fighting the desire to abandon hyperfine and create a new benchmark runner from scratch 😂

[Jason Nochlin]

I think easiest is to perform the warmup run w/o hyperfine, check the output, then use hyperfine for the 5 runs

And if we make this change, why not also do a quick run with ./test.sh to validate their implementation

[Gunnar Morling]

I think easiest is to perform the warmup run w/o hyperfine, check the output, then use hyperfine for the 5 runs

Yeah, that should go nicely together with @AlexanderYastrebov's idea above.

I keep fighting the desire to abandon hyperfine and create a new benchmark runner from scratch 😂

LOL. I kinda like it though, in particular that it displays some numbers while running.

And if we make this change, why not also do a quick run with ./test.sh to validate their implementation

That would be really neat, making everything a single invocation. And also save a bit of time, as I run mvn twice atm., once before testing and then again from evaluate2.sh.

[Jason Nochlin]

TODO:

• Change instructions that create out_expected.txt to create measurements_1B.out
• Run ./test.sh before hyperfine
• Replace hyperfine warmup with ./test.sh <fork> measurements_1B.txt and compare its result to measurements_1B.out

[Jason Nochlin]

PR Ready: #333