Test case times sometimes reported as negative
Fry-kun opened this issue · comments
Describe the bug
https://firebase-community.slack.com/messages/test-lab/thread/C1MTSQ5QT-1677888583.152469
Flank's junit xml files list some test cases like so:
<testcase name="..." classname="..." time="-0.659">
To Reproduce
- Run a test with deflaking enabled & a test case failing (triggering deflake runs)
- Check JUnitReport.xml
Expected behavior
No negative test case times
Details (please complete the following information):
This was determined to be caused by overhead time calculation introduced in #684 to fix #557. While calculating overhead, all instances of test cases are added together (which is 5*T for 4 deflake reruns), and then subtracted from T of a single run (ignoring reruns), resulting in "negative overhead"
Additional context
Notably, all results from FTL are non-negative