[Performance]: dotnet/runtime tests are bottlenecked by MSBuild performance
agocke opened this issue · comments
Issue Description
The dotnet/runtime tests consist of thousands of tests in the src/tests tree. In the lab these tests take upwards of 20 minutes to build.
However, it looks like most of this time is in MSBuild. When I build the first fraction of these tests (1100 tests) locally, it takes MSBuild 6:50s.
If I run the same projects through Roslyn's "Replay" runner, which runs the csc commands from the binlog, this takes 1:39s.
That's a roughly 5x speedup.
The overall cost here is significant complexity in the runtime build. The other piece of important info is that the libraries tests, take only a few minutes to build. That means that if we were to build the entire test tree together we would delay starting the libraries tests by > 20 minutes. To counteract this we make the build substantially more complicated by splitting out a separate job in our pipeline just to build the runtime tests. If the overall build time were significantly reduced, we could remove a lot of complexity and delay in our CI testing.
Steps to Reproduce
See dotnet/runtime src/tests tree. The command line I used to build was src/tests/build.sh allTargets skipnative skipgeneratelayout skiptestwrappers checked x64 /p:LibrariesConfiguration=Release -bl
Data
MSBuild 6:50s.
Csc 1:39s.
Analysis
No response
Versions & Configurations
No response
Regression
- yes
- no
Regression Details
No response
fyi @rainersigwald
I thought the North Star here was far fewer assemblies?
You mean fewer projects? We've already performed a significant amount of consolidation. We can continue to shrink this down over time, but it will still likely be a lot of projects. And it doesn't look like csc has a problem with this scale.
The other reason this came up is that the 1ES pipeline work revealed how extremely complicated the runtime pipeline is. It's easily the largest pipeline in dotnet, exceeding standard AzDO rate limits by two orders of magnitude. The runtime tests currently represent an unresolvable complication in our AzDO pipelines wherein we would like to remove the extra job + upload + join, but can't do so if it would regress time-to-test-results by 20 min.
You mean fewer projects? We've already performed a significant amount of consolidation. We can continue to shrink this down over time, but it will still likely be a lot of projects.
Right. I'm not defending MSBuild. But why are more than say O(100) assemblies needed? I'm just curious, not suggesting I have any special insight.
I believe it's a mix of: Native AOT tests can't be mixed because more code has trimming side effects, some tests contaminate the process such that it can't be reused, some simply have code patterns that can't easily be combined, and finally test consolidation still ends up with a fair amount of manual effort and we can only commit so much per unit time.