Flank / flank

:speedboat: Massively parallel Android and iOS test runner for Firebase Test Lab

Home Page:https://firebase.community/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Test case times sometimes reported as negative

Fry-kun opened this issue · comments

Describe the bug
https://firebase-community.slack.com/messages/test-lab/thread/C1MTSQ5QT-1677888583.152469

Flank's junit xml files list some test cases like so:

<testcase name="..." classname="..." time="-0.659">

To Reproduce

  1. Run a test with deflaking enabled & a test case failing (triggering deflake runs)
  2. Check JUnitReport.xml

Expected behavior
No negative test case times

Details (please complete the following information):
This was determined to be caused by overhead time calculation introduced in #684 to fix #557. While calculating overhead, all instances of test cases are added together (which is 5*T for 4 deflake reruns), and then subtracted from T of a single run (ignoring reruns), resulting in "negative overhead"

Additional context
Notably, all results from FTL are non-negative