CachedHashingSpec is flaky
SethTisue opened this issue · comments
I've seen it fail several times in recent CI runs.
[info] CachedHashingSpec:
[3683](https://github.com/sbt/zinc/runs/5176868906?check_suite_focus=true#step:6:3683)
[info] zinc
[3684](https://github.com/sbt/zinc/runs/5176868906?check_suite_focus=true#step:6:3684)
[info] - should cache jar generation *** FAILED ***
[3685](https://github.com/sbt/zinc/runs/5176868906?check_suite_focus=true#step:6:3685)
[info] 18 was not less than 2.5 Cache jar didn't work: 18 is >= than 50% of 5. (CachedHashingSpec.scala:91)
similar previous issue: #552 with fix #553; maybe even more leniency is needed?
To make the test more robust and also make it a better guard against performance regressions, an improved approach might be to run it a multiple times (like JMH) and compare the p50 (median). If n = 1 occasionally fails, that could mean that the improvement gap is within the variance range. But also since it's mostly passing, it also hopefully means that n doesn't have to be large to smoke out the p50 differences consistently, like n = 5, or n = 11 should be ok.
PR: #1077 (taking a somewhat simpler approach than suggested)