Flaky test suite
Lewiscowles1986 opened this issue · comments
https://github.com/Lewiscowles1986/ruby-prof/runs/6994595317?check_suite_focus=true more directly links the last passing branch from Travis, even omitting osx and windows (for-now, it's still in a matrix).
Even with that branch the CI (possibly the test-suite itself) sometimes fails, but upon re-run does not.
Seeds
- 1658
- 63686
- 2621
- 26549
Cause of failures is unknown...
Yeah some tests, especially on windows, are sometimes off by a little bit.
@cfis this seems to be affecting all the operating systems. The failures linked are all Linux.
What I Can't work out for the life of me is how the numbers could be wrong, or if perhaps the tests giving an absolute time, might cause more problems than they solve.
Is it that the timer is from the Operating system, rather than the process itself?
Just thinking to a simple example of timings
time echo hello
hello
real 0m0.003s
user 0m0.000s
sys 0m0.000s
The real-time something takes, especially in a shared computing environment, such as GitHub actions could definitely lead to skew.
Some tests measure process time and some measure wall time. I have spent a fair bit of time over the years to try and not have flaky test suite, but haven't really succeeded. Going to close since I'm not sure there is a fix.
I Really feel like this should be open with cantfix or wontfix; at the very least it should be an indicator that this cannot really be relied upon.