Check performance with/without analyzers.
AR-May opened this issue · comments
Context
When analyzers are disabled ('/analyze' not passed on command line) there should be no to minimal impact on the build perf. There should be an acceptable overhead for when the analyzers are enabled.
Goals
- Check the performance of current state (with and without analyzers).
I have checked on our automated tests in PerfStar and I do not notice any regression without the analyzers on.
As for the run of tests with analyzers, I see insignificant regression (0.4% with previous CI run and 1.4% with 30-days mean) which is on the top edge of the possible variance.
I would say it is an acceptable overhead for the new capabilities. However, not many analyzers are implemented, so it is expected that the overhead would not be high at this moment of development.
TODO: run manual perf tests on bigger repositories.
FYI @baronfel
This is an encouraging result! Can you link to the perfstar runs/charts here for posterity?
@JanKrivanek checked current main: ran msbuild on OC and console app solutions with and without analyzers.
OC rebuild | with analyzers | without analyzers |
---|---|---|
median | 85053 | 73716 |
mean | 88154 | 77364 |
console app rebuild | with analyzers | without analyzers |
median | 8982 | 8912 |
mean | 8969 | 8915 |
For small solutions overhead is small, but for bigger one it is quite noticeable.
Oh - that is almost 14% :-O
Do we have any mid or larger size solution in the batter of repos in PerfStar? It's be nice to be able to catch those slowdowns autmatically.
As for now - we'll likely need a measurement of OC build
- with
MSBUILDLOGPROPERTIESANDITEMSAFTEREVALUATION
env variable set system wide to "1" (this is required by the analyzers currently to provide data for analysis). - with the sole analyzer turned off -
IsEnabled
set tofalse
. So that we can disect the impact of infra and the analyzer.
@AR-May - can you create items and handle those?
It'd be good to test on a version without editorconfig changes (#9811) merged - as that will bring yet another big variable to the perf equation.
It might be interesting to test with #10016 (once it's finlized) - as it'd get the tracing a good test-drive (with ability to compare the results reported by TracingModule and actual 'wall clock time measurements') - FYI @maridematte (you might want to team up with @AR-May on this)
Do we have any mid or larger size solution in the batter of repos in PerfStar? It's be nice to be able to catch those slowdowns automatically.
They are currently broken; we have an issue to fix them: https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1896972
I will create items and investigate further the source of this perf issue. I agree that's a big difference.
It'd be good to test on a version without editorconfig changes (#9811) merged - as that will bring yet another big variable to the perf equation.
editorconfig changes are not yet merged, so I tested without them.
with MSBUILDLOGPROPERTIESANDITEMSAFTEREVALUATION env variable set system wide to "1"
Done, here it is. Does not explain the difference in ~10 seconds of the build time.
Time Elapsed 00:01:32.61
MSBuild.SharedOutputPathAnalyzer: 00:00:00.0213450
with the sole analyzer turned off
I turned off the analyzer and ran again with /analyze
and without. Also ran a baseline with commit fc97b2d
Results:
OC rebuild | with analyze | without analyze | baseline |
---|---|---|---|
median | 81445 | 71151 | 72182 |
mean | 82251.14286 | 73447.57143 | 71997 |
It seems like most of the perf overhead lays in the infra part and not in the only analyzer.
The baseline and run without /analyze
are same, difference is withing the variance.
With @maridematte changes:
Build succeeded.
0 Warning(s)
0 Error(s)
Time Elapsed 00:01:30.07
BuildCheck run times
Infrastructure run times
analyzerAcquisitionTime,00:00:00
analyzerSetDataSourceTime,00:00:00.0000075
newProjectAnalyzersTime,00:00:00.0001385
Analyzer run times
MSBuild.SharedOutputPathAnalyzer,00:00:00.0153347
Closing this issue - we measured the perf.
Created a new one to investigate (and hopefully fix) the perf issue.