Command line option `minimum-expected-tests` does not accept 0

[Janis]

Hi,

i tried setting --minimum-expected-tests option to 0 because my azure devops pipeline fails with a --filter on zero-tests.

https://devblogs.microsoft.com/dotnet/introducing-ms-test-runner/ mentions this:

But in the source code it says 0 is invalid:

testfx/src/Platform/Microsoft.Testing.Platform/CommandLine/PlatformCommandLineProvider.cs

Lines 130 to 134 in 86489a2

 private static Task<ValidationResult> IsMinimumExpectedTestsOptionValidAsync(CommandLineOption option, string[] arguments) 
     => option.Name == MinimumExpectedTestsOptionKey 
         && (arguments.Length != 1 || !int.TryParse(arguments[0], out int value) || value == 0) 
         ? ValidationResult.InvalidTask(PlatformResources.PlatformCommandLineMinimumExpectedTestsOptionSingleArgument) 
         : ValidationResult.ValidTask; 

For now i added --ignore-exit-code 8 as a workaround. Is there a better way?

[Amaury Levé]

There is definitely something wrong. I am not sure to understand the use case where you would want to run no test, it would seem like a bug to me.

[Marco Rossignoli]

For now i added --ignore-exit-code 8 as a workaround. Is there a better way?

This is correct.

I think we there's a bug in the documentation. Run 0 tests is unexpected so imo doesn't make a lot of sense have the idea of "minimum" on 0 quantity.

[Janis]

Thank you!

[SymbioticKilla]

There is definitely something wrong. I am not sure to understand the use case where you would want to run no test, it would seem like a bug to me.

Hi,
we use normally --filter to remove integration tests. With old MSTest works without problem.
In our case it is "dotnet test solution.sln with -p:TestingPlatformCommandLineArguments" like here #2175
MSTest Runner throws error because integration tests report 0 tests. Actually --filter should ignore them completely?

[Amaury Levé]

MSTest Runner throws error because integration tests report 0 tests. Actually --filter should ignore them completely?

I am not sure to understand what you mean. With the runner (compared to VSTest) we have made the choice to be strict by default and we do consider running no test as an abnormal situation (this was a recurring complaint in VSTest). If you think this is fine for your situation to request test execution of zero tests, then you should use the --ignore-exit-code.

Let me know if I am missing something.

[SymbioticKilla]

MSTest Runner throws error because integration tests report 0 tests. Actually --filter should ignore them completely?

I am not sure to understand what you mean. With the runner (compared to VSTest) we have made the choice to be strict by default and we do consider running no test as an abnormal situation (this was a recurring complaint in VSTest). If you think this is fine for your situation to request test execution of zero tests, then you should use the --ignore-exit-code.

Let me know if I am missing something.

Thanks for quick answer.
Example:
Solution has 10 test projects. One of the projects is integration tests project.
We want to run only unit tests projects with "dotnet test Solution.sln..." So we use --filter parameter to filter integration tests projects out.
Short story:
Step 1: Run unit tests from solution
Step 2: Run integration tests from solution later in the build pipeline