Dotnet test. Add ability to run test projects in parallel

[Anton Smolkov]

Hi! It would be super useful if dotnet test step had the ability to run tests in parallel.

Actually, sincedotnet sdk 2.1 it's possible to pass .SLN file to dotnet test instead of .csproj and use xunit /p:ParallelizeAssemblies msbuild parameterer, but this approach has drawbacks for legacy .NET framework projects.
Dotnet SDK has only limited support of legacy .net projects. For instance, it does not work, if test project references legacy asp.net project and fails on import of legacy-specific msbuild targets

[Nikolay Pianikov]

@AnSmol I am no sure I understand. What is hepening when you run xunit tests using dotnet test xxx.sln with additional argument /p:ParallelizeAssemblies now? Are they not working as expected?

TeamCity just runs command line tool dotnet to run tests and it tracks tests results. Do you suppose that TeamCity should run several dotnet processes at the same time or something else?

[Anton Smolkov]

What is hepening when you run xunit tests using dotnet test xxx.sln with additional argument /p:ParallelizeAssemblies now? Are they not working as expected?

@NikolayPianikov ParallelizeAssemblies It works as expected, but one .csproj = one test assembly, so effectively it is not parallelization at all. I guess it suppose to work only on solution level.

Do you suppose that TeamCity should run several dotnet processes at the same time ..?

Exactly! Ability to define parallelization level also would be great.

[Anton Smolkov]

It's appeared that dotnet test on .SLN runs test .csproj'es on parallel by default, so i don't even have to append extra flags for xunit.

While investicage all this stuff found following drawbacks of the teamcity-dotnet-plugin step:

• Lates version of .NET CLI Support plugin ( 0.9.7 ) still appends -verbosity Normal instead of -verbosity normal. Issue. I know the problem is on the VStest's side, but still.

• Can't pass /p:VSToolsPath="%MSBuildTools16.0_x64_Path%\..\..\..\Microsoft\VisualStudio\v16.0" as a custom parameter, it converts to "/p:VSToolsPath="%MSBuildTools16.0_x64_Path%\..\..\..\Microsoft\VisualStudio\v16.0"" (extra double quotes) I guess it's because of TW-7964. I would pass this property as a teamcity build parameter, but i don't want it on my other msbuld step which makes build.

[Anton Smolkov]

if someone is interested, the complete way to succesfully run dotnet test on .SLN file (and thus in parallel) containing different kind of projects (Legacy CsProj/ProjectSDK, Legacy Asp.Net/Asp.Net Core, Xunit/Nunit):

1. Inside test projects, add nuget-references to Microsoft.NET.Test.Sdk, TeamCity.VSTest.TestAdapter and test-platform specific adapters. As described here

2. For passing dotnet test validation of tested legacy asp.net projects either add /p:VSToolsPath="%MSBuildTools16.0_x64_Path%\..\..\..\Microsoft\VisualStudio\v16.0" parameter to run or use MSBuild.Microsoft.VisualStudio.Web.targets nuget package inside legacy Asp.Net project.
   Both are possible because of default import in leacy asp.net .csproj'es:
   <Import Project="$(VSToolsPath)\WebApplications\Microsoft.WebApplication.targets" Condition="'$(VSToolsPath)' != ''"/>

3. Build solution and run dotnet test with --no-build --verbosity normal. (Verbosity is xunit-specific and must be spelled in lowercase)

So I personally ended up with the following custom powershell script step:

Set-PSDebug -Trace 1
#Have to run via script because of TW-7964
#VSToolsPath overrided to vs build tools one, to make tests on legacy asp.net projects work
#To make adapters service messages work, verbocity must be set as normal or higher, spelled in lovercase
&"%DotNetCLI_Path%" --% test %SLN_FILE% --configuration Release --no-build --verbosity normal /p:VSToolsPath="%MSBuildTools16.0_x64_Path%\..\..\..\Microsoft\VisualStudio\v16.0"
exit $LASTEXITCODE 

[Nikolay Pianikov]

@AntonSmolkov You have made a good investigation. Could you share this information in some section of readme? Some git repo with samples could be very useful too :)