New MSTest runner doesn' t integrate with coverlet
MarkusRodler opened this issue · comments
Previously I used dotnet test /p:CollectCoverage=true with coverlet.msbuild to produce an coverage output on the command line like that:
+--------+------+--------+--------+
| Module | Line | Branch | Method |
+--------+------+--------+--------+
| Maui | 0% | 100% | 0% |
+--------+------+--------+--------+
| Shared | 0% | 0% | 0% |
+--------+------+--------+--------+
| Client | 0% | 100% | 0% |
+--------+------+--------+--------+
| Test | 0% | 0% | 0% |
+--------+------+--------+--------+
+---------+------+--------+--------+
| | Line | Branch | Method |
+---------+------+--------+--------+
| Total | 0% | 0% | 0% |
+---------+------+--------+--------+
| Average | 0% | 50% | 0% |
+---------+------+--------+--------+But with the new MS Test runner this isn't possible anymore. It would be nice that this feature is restored. Or if someone has an alternative that shows the coverage for each module.
The new testing platform upon which the MSTest runner is build has got different extensibility point and MSBuild integration.
You can open an issue on coverlet repo for it https://github.com/coverlet-coverage/coverlet/issues
You can use the coverlet tool for now https://github.com/coverlet-coverage/coverlet?tab=readme-ov-file#net-global-tool-guide-suffers-from-possible-known-issue or move to the MS coverage version that's already integrated https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-platform-extensions-code-coverage#microsoft-code-coverage
With the MSTest.Sdk trx and codecoverage are already opted in by default https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-mstest-sdk#mstest-runner-profile
Here the documentation https://github.com/microsoft/codecoverage/blob/main/samples/Algorithms/scenarios/scenario07/README.md
Thanks for the reply!
You can use the coverlet tool for now https://github.com/coverlet-coverage/coverlet?tab=readme-ov-file#net-global-tool-guide-suffers-from-possible-known-issue
That works
or move to the MS coverage version that's already integrated https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-platform-extensions-code-coverage#microsoft-code-coverage
That works but does not give the coverage output on the command line.
With the MSTest.Sdk trx and codecoverage are already opted in by default https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-mstest-sdk#mstest-runner-profile
I think TRX is not useful for this scenario.
That works but does not give the coverage output on the command line.
Only the msbuild integration scenario of Coverlet shows information in console, usually ppl that are not using it are generating a report with tools like report generator https://github.com/danielpalme/ReportGenerator
@fhnaseer @jakubch1 is there some plan to have the coverage output on console?
I find the coverage output on the command line very useful. It works with the coverlet tool but as far as I understand you have to first run the tests with dotnet run to generate a dll. And then you can run the coverlet tool with that dll. This means that I can't run the tool with dotnet watch to always run the tests and output the current coverage. Even not if I make a msbuild task for that because msbuild doesn't work with dotnet watch.
Console output is useful but doesn't scale for big projects, usually IDEs offers contextual report like Visual Studio as explained here https://devblogs.microsoft.com/visualstudio/code-coverage-features-in-visual-studio-enterprise/
This means that I can't run the tool with dotnet watch to always run the tests and output the current coverage.
Are you using the hot reload extension https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-platform-extensions-hosting#hot-reload?
Yes that doesn't scale for big projects but we use microservices and that fits better.
Are you using the hot reload extension https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-platform-extensions-hosting#hot-reload?
Not yet
With the MSTest.Sdk trx and codecoverage are already opted in by default https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-mstest-sdk#mstest-runner-profile
Here the documentation https://github.com/microsoft/codecoverage/blob/main/samples/Algorithms/scenarios/scenario07/README.md
How does this work with running multiple test projects in a solution? Before using the MSTest Runner, which I'm trying with MSTest.Sdk, I would use dotnet test with a run settings file or command line arguments.
I have struggled to find the "solution wide coverage" approach. I'd be happy with multiple files, one per project, in a central directory. Merging is trivial with tools like report generator (which ADO pipelines use).
Edit
Well, sometimes typing it out helps. It seems MSTest.Sdk is the issue, because I have it working without that, but using MSTestRunner. Essentially, if I follow the "smaller" csproj as recommended in the Enable MSTest runner in a MSTest project docs, then solution wide coverage breaks for me. I may open a new ticket, but for now I have my solution: Do not use MSTest.Sdk.
Also -- yes -- I definitely am misunderstanding pieces of this. I have found it very difficult to understand what works with what given some "active" documentation predates this new way of working.
I have struggled to find the "solution wide coverage" approach.
If you've a solution with all MSTest runner configuration(MStest.Sdk) there's no automatic way to merge at solution level with dotnet test. We've planned to support it.
So the solution is to merge at the end in a custom step using the dotnet-coverage tool (dotnet-coverage merge);
here you can find some sample:
https://github.com/microsoft/codecoverage/blob/main/samples/Calculator/scenarios/scenario18/README.md
https://github.com/microsoft/codecoverage/blob/main/samples/Algorithms/README.md