Reporting Mechanism for the kill Matrix

[Nassim Maluli]

I am currently working on a Flutter testing project to implement effective mutation testing techniques. However, these techniques require a kill matrix that can illustrate which test kills which mutant. I was wondering if there is any way to detect this using built-in mechanisms of the tool

[domohuhn]

Currently, no. The generated report will only show the mutations, not the test case. The stats are also only gathered per test command and the mutations are not tracked, just an overall count. I think you can only increase the level of detail in the report to test cases in the following scenarios:

• Each test case is listed as individual command in the input xml file. That would still require code changes to track the combination of failed test and command. This requires an ahead of time discovery of the test cases. dart test and flutter test both do not support listing all test cases.
• there is a well defined output format that can be parsed, showing the failed test cases. Maybe the json reporter could be used for this, but the test package documentation specifically states that the format is part of the API and may change with different versions of the test package. This would introduce a hard dependency on the version of the dart test package. Every time a new test package version is released, this tool has to be tested and maybe adapted. https://github.com/dart-lang/test/blob/master/pkgs/test/doc/json_reporter.md

I am not sure what you actually want to gain by using this information? Do you want to thin out the test cases for future runs? I don't think that this is something you should aim for. You may be able to detect "bad" tests that do not detect any mutation/do not contain asserts, but that should be covered via code review (at least if you care about testing and the correctness of your code...). Removing mutations also runs into problems - if you modify the code that was tested by these mutations, you may miss problems that were introduced by the new commits.

I think the best way to increase the efficiency is to run the mutation tests only on the diff between commits:
https://github.com/domohuhn/mutation-test#running-an-incremental-analysis-ci

[Nassim Maluli]

Interesting !, Thank you !