Collected data seems suspicious; Canary mode doesn't report metrics
jimfb opened this issue · comments
I checked out the demo and ran it with the following results:
forcedGcAmount | forcedGcTime | gcAmount | gcTime | majorGcTime | pureScriptTime | renderTime | scriptTime
------------------ | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ | ------------------
1299.77 | 13.06 | 491.90 | 10.73 | 0.00 | 51.52 | 134.63 | 62.25
2367.62 | 29.73 | 0.00 | 0.00 | 0.00 | 73.11 | 138.69 | 73.11
2146.84 | 28.58 | 1400.20 | 6.02 | 0.00 | 64.64 | 138.59 | 70.66
3729.60 | 33.67 | 0.00 | 0.00 | 0.00 | 64.24 | 141.94 | 64.24
3765.45 | 34.57 | 0.00 | 0.00 | 0.00 | 65.50 | 137.77 | 65.50
3775.76 | 39.67 | 0.00 | 0.00 | 0.00 | 67.22 | 135.41 | 67.22
3769.90 | 37.77 | 0.00 | 0.00 | 0.00 | 66.30 | 142.54 | 66.30
3776.86 | 35.73 | 0.00 | 0.00 | 0.00 | 70.08 | 146.30 | 70.08
3761.22 | 37.83 | 0.00 | 0.00 | 0.00 | 67.39 | 139.36 | 67.39
3762.50 | 37.86 | 0.00 | 0.00 | 0.00 | 64.04 | 148.04 | 64.04
3794.86 | 38.85 | 0.00 | 0.00 | 0.00 | 63.96 | 135.96 | 63.96
3781.95 | 38.89 | 0.00 | 0.00 | 0.00 | 65.23 | 143.53 | 65.23
3764.52 | 38.47 | 0.00 | 0.00 | 0.00 | 67.48 | 144.74 | 67.48
3759.41 | 36.89 | 0.00 | 0.00 | 0.00 | 66.44 | 137.67 | 66.44
3762.62 | 36.82 | 0.00 | 0.00 | 0.00 | 67.38 | 142.58 | 67.38
3774.42 | 41.91 | 0.00 | 0.00 | 0.00 | 65.47 | 138.46 | 65.47
3764.98 | 40.21 | 0.00 | 0.00 | 0.00 | 66.89 | 140.60 | 66.89
3759.39 | 41.70 | 0.00 | 0.00 | 0.00 | 66.40 | 144.26 | 66.40
3755.74 | 36.05 | 0.00 | 0.00 | 0.00 | 71.11 | 137.41 | 71.11
3764.54 | 37.54 | 0.00 | 0.00 | 0.00 | 69.03 | 143.31 | 69.03
================== | ================== | ================== | ================== | ================== | ================== | ================== | ==================
3491.90+-19% | 35.79+-17% | 94.60+-336% | 0.84+-312% | 0.00 | 66.17+-6% | 140.59+-2% | 67.01+-3%
.
The GC times looked awfully suspicious to me (but I wasn't surprised, since the runner appeared to be running the default version of chrome). I modified chromeOptions to include 'binary': '/Applications/Google\ Chrome\ Canary.app/Contents/MacOS/Google\ Chrome\ Canary',, which did successfully cause Canary to be launched, but then the results were empty:
forcedGcAmount | forcedGcTime | gcAmount | gcTime | majorGcTime | pureScriptTime | renderTime | scriptTime
------------------ | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ | ------------------
0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.26 | 0.00
0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.27 | 0.00
0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.61 | 0.00
0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.46 | 0.00
0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.22 | 0.00
0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.44 | 0.00
0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.57 | 0.00
0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.36 | 0.00
0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.31 | 0.00
0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.28 | 0.00
0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.52 | 0.00
0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.37 | 0.00
0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.61 | 0.00
0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.43 | 0.00
0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.53 | 0.00
0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.37 | 0.00
0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.28 | 0.00
0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.62 | 0.00
0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.41 | 0.00
0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.41 | 0.00
================== | ================== | ================== | ================== | ================== | ================== | ================== | ==================
0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.42+-29% | 0.00
I'm new to angular / protractor / benchpress, so maybe I'm missing something obvious. Any ideas?
@jimfb did you modify your tests at all between samples? Seems as if test isn't running as expected. You may also want to start up a server and manually interact with the application, then inspect the console to see if any errors are reported.
@jeffbcross Nope, didn't modify the tests. Clean checkout of this repository, only change is that on the second run I set the chromeOptions binary path (and saw that chrome canary did run, so the path is correct). I also saw the webdriver clicking the tests, so it appeared to be working, and it doesn't appear to be a complete failure because the renderTime column has data.
I just tried starting the server manually and running the test myself in chrome and chrome-canary, both looked perfectly reasonable to me.
Is it possible you could share the complete runnable app, benchmark, setup, etc?
It's literally just a clean checkout for the first run, and a single line ('binary': '/Applications/Google\ Chrome\ Canary.app/Contents/MacOS/Google\ Chrome\ Canary',) added to protractor.conf.js for the second run. I just ran the three commands in the wiki and pasted the results here.
Other than that, the only relevant setup is my chrome&canary installations (both standard, afaik) and my OS (Yosemite 10.10.4 (14E46)) - I'm not really sure how to "share" those. What did you have in mind?
Sounds like the same issue as reported here: angular/angular#3411
@jimfb do you want to take a stab at a PR to fix benchpress?