Running Unit test is taking very long time

Question

Running Unit test is taking very long time

maartenmensink opened this issue 7 years ago · comments

We run NUnit test with dotnet test throught the lastest.

It is taking ages before the unit test run on the build server while locally the unit test run within 7 seconds. The build server takes at least 120 seconds pulse before even running the first unit test.

After some digging around we located a .rsp file which contained a path the VsTestTestApaterPath which is added by the plugin to the working directory in a folder with a guid as foldername.

The path in the .rsp file
/p:VSTestTestAdapterPath="C:\BuildAgent\work\e34cbcfda0481938"

But the actual location of the test adapter was

C:\BuildAgent\work\e34cbcfda0481938\21c30b741f184828b6d2f856e377a906

When running the command on a cmd window on the build agent we got the same result as was showing on the build server.

C:\Program Files\dotnet\dotnet.exe" test C:\BuildAgent\work\e34cbcfda0481938\Test\Project.Test\Project.Test.csproj --configuration Release --no-build @C:\BuildAgent\temp\buildTmp\43ac88685c004c409c8c719e5e354013.rsp

After updating the /p:VSTestTestAdapterPath param in the .rsp file to /p:VSTestTestAdapterPath="C:\BuildAgent\work\e34cbcfda0481938\21c30b741f184828b6d2f856e377a906" and running the same command again the command ran within 7 seconds.

Dmitry Tretyakov · Answer 1 · Wed Oct 25 2017 23:00:53 GMT+0800 (China Standard Time)

@maartenmensink, could you please also report .NET CLI plugin version?

Maarten Mensink · Answer 2 · Wed Oct 25 2017 23:03:15 GMT+0800 (China Standard Time)

@dtretyakov SNAPSHOT-20171010161822

Alex Allen · Answer 3 · Fri Oct 27 2017 21:53:42 GMT+0800 (China Standard Time)

I'm experiencing very slow unit test execution from this plugin too:

.NET CLI plugin version: SNAPSHOT-20171027110132
TeamCity: 2017.1.3 (build 46961)

[14:43:58]Build completed.
[14:43:58]
[14:43:58]Test run for D:\TeamCity\buildAgent2\work\...\tests\UnitTests\bin\Debug\netcoreapp2.0\UnitTests.dll(.NETCoreApp,Version=v2.0)
[14:43:59]Microsoft (R) Test Execution Command Line Tool Version 15.3.0-preview-20170628-02
[14:43:59]Copyright (c) Microsoft Corporation. All rights reserved.
[14:43:59]
[14:44:00]Starting test execution, please wait...
[14:48:27][xUnit.net 00:00:47.8647078]

There is nearly 4 minutes of silence after 'starting test execution' until xUnit is start it's counting.
Was previously using version 0.8 (I think) of the plugin, and this was definitely not a problem then.

Maarten Mensink · Answer 4 · Fri Nov 03 2017 20:23:35 GMT+0800 (China Standard Time)

@dtretyakov any update on this?

Dmitry Tretyakov · Answer 5 · Sat Nov 04 2017 00:24:56 GMT+0800 (China Standard Time)

@maartenmensink, we're investigating the cause of it. Since version 0.8 was added support of vstest listener which could affect performance.

Nikolay Pianikov · Answer 6 · Mon Nov 13 2017 21:42:57 GMT+0800 (China Standard Time)

Is it possible to share some minimal solution to reproduce?

Maarten Mensink · Answer 7 · Thu Nov 16 2017 22:02:06 GMT+0800 (China Standard Time)

@NikolayPianikov We will try to create minimal reproducible solution.

Nikolay Pianikov · Answer 8 · Fri Nov 17 2017 18:08:32 GMT+0800 (China Standard Time)

We have the following cases:
1 - .net core projects
2 - .net full framework projects

TeamCity

should add its custom logger/adapter to provide on the-fly-test reporting via TeamCity service messages
should not hide any tests specific adapters (xunit/mstest and so on)

Case 1: test host for .net core tries finding loggers by the name *TestLogger.dll or *TestAdapter.dll:

in the bin/restore path
in the path specified by VSTestTestAdapterPath and all its child directories

Case 2: test host for full .net:

in the bin/restore path IF VSTestTestAdapterPath IS NOT SPECIFIED
in the path specified by VSTestTestAdapterPath and all its child directories

So for general case (when we have all types of project) TeamCity:

creates temporary directory in the TeamCity agent's checkout directory
copys TeamCity logger to it before each test step
overrides VSTestTestAdapterPath as a path to TeamCity agent's checkout directory

So vstest could spend potentially significant time finding all required adapters when agent checkout directory has a lot of files/directories.

@maartenmensink could you share your tree for agent's checkout directory including list of files for example by command dir /b /s > abc.txt

JanKolk · Answer 9 · Fri Nov 17 2017 20:24:08 GMT+0800 (China Standard Time)

@NikolayPianikov We figured out what had impact on the test time. We have 2 angular projects with NPM packages.

Result with 2 angular projects with downloaded NPM packages

[13:04:07][test] Starting test execution, please wait...
[13:05:34][test] NUnit Adapter 3.9.0.0: Test execution started

Result after deleting the 2 projects with all NPM packages

[13:10:46][test] Starting test execution, please wait...
[13:10:49][test] NUnit Adapter 3.9.0.0: Test execution started

Directory tree

src
-- Projects
Test
-- Test project one
UI
-- First UI project
--- node_modules
--- other files
-- Second UI project
--- node_modules
--- other files

Project filter
Test\**\*.Test.csproj

I hope this information helps.

Nikolay Pianikov · Answer 10 · Fri Nov 17 2017 20:48:17 GMT+0800 (China Standard Time)

@JanKolk thank you.

We decided to add a configuration parameter to switch off logger's injection here

In this case you could add this package reference to have a test reporting.

In the future we a going to find other way

Phil Tremblay · Answer 11 · Fri Dec 08 2017 03:46:01 GMT+0800 (China Standard Time)

Hi there

This problem seems worst when we plug dotcover into it. I've got the same testing build step but one has dotcover configured and this config takes 3-4 times longer (3 minutes to 12).

Maybe it is because with dotcover activated, we use the Assembly filters who might search for Adapter everywhere.

@NikolayPianikov where do we set the dotnet.cli.test.reporting=off settings? Is it a build step parameter or agent configuration?

Thanks
Phil

Nikolay Pianikov · Answer 12 · Fri Dec 08 2017 03:56:43 GMT+0800 (China Standard Time)

@jesuissur it is configuration parameter in a build, see this page

yes dotCover significantly decreases a tests performance

Phil Tremblay · Answer 13 · Fri Dec 08 2017 04:16:16 GMT+0800 (China Standard Time)

Yes, dotCover is longer than without coverage but here the steep rise we got with the same build before and after the latest TeamCity upgrade (2017.2)

Nikolay Pianikov · Answer 14 · Fri Dec 08 2017 04:20:23 GMT+0800 (China Standard Time)

@jesuissur could you attach build log files before and after upgrade?

Phil Tremblay · Answer 15 · Fri Dec 08 2017 05:28:51 GMT+0800 (China Standard Time)

I'll start with a build log sample. If you'd like more, just let me know.

Before

[Step 5/15] dotnet.exe test C:\DevTools\TeamCity\buildAgent\work\4058df1f286aadf\test\ThisCompany.Core.Mvc.Tests\ThisCompany.Core.Mvc.Tests.csproj @C:\DevTools\TeamCity\buildAgent\temp\agentTmp\2cd92c20bc654fb7b96782f631b2a0e2.rsp
[11:01:34][Step 5/15] test (42s)
[11:01:34][test] Starting: C:\DevTools\TeamCity\buildAgent\tools\JetBrains.dotCover.CommandLineTools.2017.3.0-eap07\dotCover.exe cover C:\DevTools\TeamCity\buildAgent\temp\buildTmp\5924c273a55e4e91b103f257cb673e5e.dotCover /ReturnTargetExitCode /NoCheckForUpdates /AnalyzeTargetArguments=false
[11:01:34][test] in directory: C:\DevTools\TeamCity\buildAgent\work\4058df1f286aadf
[11:01:34][test] JetBrains dotCover Console Runner 2017.3 EAP 7. Build 111.0.20171123.173629-eap07
[11:01:34][test] Copyright (c) 2009-2017 JetBrains s.r.o. All rights reserved.
[11:01:34][test] [JetBrains dotCover] Coverage session started [11/29/2017 11:01:34 AM]
[11:01:39][test] Build started, please wait...
[11:01:47][test] Build completed.
[11:01:47][test]
[11:01:47][test] Test run for C:\DevTools\TeamCity\buildAgent\work\4058df1f286aadf\test\ThisCompany.Core.Mvc.Tests\bin\Debug\netcoreapp1.1\ThisCompany.Core.Mvc.Tests.dll(.NETCoreApp,Version=v1.1)
[11:01:48][test] Microsoft (R) Test Execution Command Line Tool Version 15.0.0.0
[11:01:48][test] Copyright (c) Microsoft Corporation. All rights reserved.
[11:01:48][test]
[11:01:49][test] Starting test execution, please wait...
[11:02:13][test] [xUnit.net 00:00:21.6213592]
...
[11:02:14][test] [xUnit.net 00:00:22.3785397]
[11:02:14][test]
[11:02:14][test] Total tests: 6. Passed: 6. Failed: 0. Skipped: 0.
[11:02:14][test] Test Run Successful.
[11:02:14][test] Test execution time: 24.9743 Seconds
[11:02:14][test]
[11:02:14][test]
[11:02:14][test]
[11:02:14][test] Build succeeded.
[11:02:14][test] 0 Warning(s)
[11:02:14][test] 0 Error(s)
[11:02:14][test]
[11:02:14][test] Time Elapsed 00:00:36.35
[11:02:14][test]
[11:02:16][test] [JetBrains dotCover] Coverage session finished [11/29/2017 11:02:16 AM]
[11:02:16][test] [JetBrains dotCover] Coverage results post-processing started [11/29/2017 11:02:16 AM]
[11:02:16][test] [JetBrains dotCover] Coverage results post-processing finished [11/29/2017 11:02:16 AM]
[11:02:16][test] Process exited with code 0

After

[Step 5/15] dotnet.exe test C:\TeamCityAgentTemp\work\4058df1f286aadf\test\ThisCompany.Core.Mvc.Tests\ThisCompany.Core.Mvc.Tests.csproj @C:\TeamCityAgentTemp\temp\agentTmp\4a5f15a349c041fdafd7a8af7d09d343.rsp
[13:02:13][Step 5/15] test (3m:55s)
[13:02:14][test] Starting: C:\DevTools\TeamCity\buildAgent\tools\JetBrains.dotCover.CommandLineTools.2017.3.0-eap07\dotCover.exe cover C:\TeamCityAgentTemp\temp\buildTmp\ae0b8222d3474889be2c51ac3489f89c.dotCover /ReturnTargetExitCode /NoCheckForUpdates /AnalyzeTargetArguments=false
[13:02:14][test] in directory: C:\TeamCityAgentTemp\work\4058df1f286aadf
[13:02:14][test] JetBrains dotCover Console Runner 2017.3 EAP 7. Build 111.0.20171123.173629-eap07
[13:02:14][test] Copyright (c) 2009-2017 JetBrains s.r.o. All rights reserved.
[13:02:14][test] [JetBrains dotCover] Coverage session started [11/30/2017 1:02:14 PM]
[13:02:28][test]
[13:02:28][test] Build succeeded.
[13:02:28][test] 0 Warning(s)
[13:02:28][test] 0 Error(s)
[13:02:28][test]
[13:02:28][test] Time Elapsed 00:00:05.73
[13:02:28][test]
[13:02:31][test] Build started, please wait...
[13:02:39][test] Build completed.
[13:02:39][test]
[13:02:39][test] Test run for C:\TeamCityAgentTemp\work\4058df1f286aadf\test\ThisCompany.Core.Mvc.Tests\bin\Debug\netcoreapp1.1\ThisCompany.Core.Mvc.Tests.dll(.NETCoreApp,Version=v1.1)
[13:02:40][test] Microsoft (R) Test Execution Command Line Tool Version 15.3.0-preview-20170628-02
[13:02:40][test] Copyright (c) Microsoft Corporation. All rights reserved.
[13:02:40][test]
[13:02:41][test] Starting test execution, please wait...
[13:06:05][test] [xUnit.net 00:00:20.4831558]
...
[13:06:06][test] [xUnit.net 00:00:21.1813673]
[13:06:06][test]
[13:06:06][test] Total tests: 6. Passed: 6. Failed: 0. Skipped: 0.
[13:06:06][test] Test Run Successful.
[13:06:06][test] Test execution time: 25.2114 Seconds
[13:06:07][test]
[13:06:07][test] Build succeeded.
[13:06:07][test] 0 Warning(s)
[13:06:07][test] 0 Error(s)
[13:06:07][test]
[13:06:07][test] Time Elapsed 00:03:36.73
[13:06:07][test]
[13:06:08][test] [JetBrains dotCover] Coverage session finished [11/30/2017 1:06:08 PM]
[13:06:08][test] [JetBrains dotCover] Coverage results post-processing started [11/30/2017 1:06:08 PM]
[13:06:09][test] [JetBrains dotCover] Coverage results post-processing finished [11/30/2017 1:06:09 PM]
[13:06:09][test] Process exited with code 0

Phil Tremblay · Answer 16 · Fri Dec 08 2017 05:35:44 GMT+0800 (China Standard Time)

@NikolayPianikov As soon as I added the dotnet.cli.test.reporting parameter, it went back to the normal running time. 👍 for the workaround

Dmitry Tretyakov · Answer 17 · Tue Dec 12 2017 19:32:02 GMT+0800 (China Standard Time)

Another possible workaround is to use vstest command instead of test since it supports more precise test adapter path declaration.

Maarten Mensink · Answer 18 · Thu Dec 14 2017 19:23:14 GMT+0800 (China Standard Time)

closing this work around available

Nikolay Pianikov · Answer 19 · Fri Mar 23 2018 17:43:22 GMT+0800 (China Standard Time)

.NET Core SDK 2.1.102 has fix allowing to improve tests performance

Update Visual Studio to latest or install dotnet CLI 2.1.102 or later
Update this plugin

Phil Tremblay · Answer 20 · Fri Mar 23 2018 20:14:44 GMT+0800 (China Standard Time)

For sure it is better.
One of our build looks like that after the updates