change 'Tests Run' and 'Passed' to reflect the number of testcases instead of the number of assertions
peetkes opened this issue · comments
In the GUI the Suite overview gives
- Total Test Count, reflects the number of testcases
- Tests Run, reflects the number of assertions
- Passed, reflects the number of successful assertions
- Failed, the number of failed testcases
A suite can have multiple testcases and each testcase can have multiple assertions.
But as soon as a testcase has a single failed assertion, the whole testcase is flagged as failed regardless of the number of succeeded assertions up until the failure.
So the count of assertions is not that informative.
I would suggest changing the 'Tests Run' to the number of testcases that were executed and 'Passed' to the number of successful run testcases.
This can be changed in test-controller.xqy lines 183, 184 and 185
attribute total { fn:count($results/test:test) },
attribute failed { fn:count($results/test:test[test:result[@type='fail']]) },
attribute passed { fn:count($results/test:test[test:result[@type='success']]) },
"number of testcases" = the number of files, right? And "Total Test Count" is already capturing that.
What marklogic-unit-test calls a test:test is really an assertion. I think the data makes sense (ignoring other issues about e.g. setup failures counting as failed assertions), we need better labels. I think these would be better labels:
- "Total Test Count" -> "Test Count" or "Total Tests" = the number of test modules (this is currently including setup files, which is a bug)
- "Tests Run" -> this could arguably go away, as it's just summing the number of successful and failed assertions. If we keep it, it's "Assertion Count" or "Total Assertions".
- "Passed" -> "Success Count" or "Succeeded" (could do "Passed", though it bothers me that we're then using "success" and "passed" instead of just one term; I guess to minimize changes, we'd keep this as "Passed").
- "Failed" -> "Failure Count" or "Failed".