Slower than just running in sequence
brandonros opened this issue · comments
Running "karma:source" (karma) task
18 01 2018 23:41:26.613:INFO [framework:karma-parallel]: sharding specs across 4 browsers
18 01 2018 23:41:27.591:INFO [karma]: Karma v0.13.22 server started at https://localhost:9876/
18 01 2018 23:41:27.597:INFO [launcher]: Starting browser Chrome
18 01 2018 23:41:27.618:INFO [launcher]: Starting browser Chrome
18 01 2018 23:41:27.624:INFO [launcher]: Starting browser Chrome
18 01 2018 23:41:27.632:INFO [launcher]: Starting browser Chrome
18 01 2018 23:41:37.012:INFO [Chrome 63.0.3239 (Mac OS X 10.12.6)]: Connected on socket uiaNpBrlnoz7lB4bAAAA with id 56325298
18 01 2018 23:41:37.172:INFO [Chrome 63.0.3239 (Mac OS X 10.12.6)]: Connected on socket NTZxRMD7CKRMXZy5AAAC with id 99647674
18 01 2018 23:41:37.277:INFO [Chrome 63.0.3239 (Mac OS X 10.12.6)]: Connected on socket SoejE-KmEwBQqQKfAAAD with id 88877803
18 01 2018 23:41:37.287:INFO [Chrome 63.0.3239 (Mac OS X 10.12.6)]: Connected on socket 9u2Jwcy1NAahSKL1AAAB with id 81113574
Chrome 63.0.3239 (Mac OS X 10.12.6): Executed 560 of 1075 SUCCESS (0 secs / 52.713 secs)
Chrome 63.0.3239 (Mac OS X 10.12.6): Executed 567 of 1075 SUCCESS (0 secs / 52.832 secs)
Chrome 63.0.3239 (Mac OS X 10.12.6): Executed 567 of 1075 SUCCESS (0 secs / 52.832 secs)
Chrome 63.0.3239 (Mac OS X 10.12.6): Executed 602 of 1075 SUCCESS (0 secs / 53.641 secs)
Chrome 63.0.3239 (Mac OS X 10.12.6): Executed 612 of 1075 SUCCESS (0 secs / 53.907 secs)
Chrome 63.0.3239 (Mac OS X 10.12.6): Executed 627 of 1075 SUCCESS (0 secs / 54.26 secs)
Chrome 63.0.3239 (Mac OS X 10.12.6): Executed 713 of 1075 SUCCESS (0 secs / 56.603 secs)
Chrome 63.0.3239 (Mac OS X 10.12.6): Executed 792 of 1075 SUCCESS (0 secs / 57.37 secs)
Chrome 63.0.3239 (Mac OS X 10.12.6): Executed 844 of 1075 SUCCESS (0 secs / 58.078 secs)
Chrome 63.0.3239 (Mac OS X 10.12.6): Executed 850 of 1075 SUCCESS (0 secs / 58.153 secs)
Chrome 63.0.3239 (Mac OS X 10.12.6): Executed 1075 of 1075 SUCCESS (1 min 2.914 secs / 1 min 2.092 secs)
Chrome 63.0.3239 (Mac OS X 10.12.6): Executed 1075 of 1075 SUCCESS (1 min 4.343 secs / 1 min 3.21 secs)
Chrome 63.0.3239 (Mac OS X 10.12.6): Executed 1075 of 1075 SUCCESS (59.78 secs / 59.114 secs)
Chrome 63.0.3239 (Mac OS X 10.12.6): Executed 1075 of 1075 SUCCESS (1 min 2.33 secs / 1 min 1.078 secs)
TOTAL: 4300 SUCCESS
I also would have expected the 1,075 tests (123 matches of describe() across 122 files) to have been divided by 4 and then executed across the browsers.
I wonder if there's a cool/fast way to help speed up the loading of 4 browsers. Maybe one browser, then 4 tabs within it?
Compare it to:
Running "karma:source" (karma) task
18 01 2018 23:45:02.811:INFO [karma]: Karma v0.13.22 server started at https://localhost:9876/
18 01 2018 23:45:02.821:INFO [launcher]: Starting browser Chrome
18 01 2018 23:45:06.159:INFO [Chrome 63.0.3239 (Mac OS X 10.12.6)]: Connected on socket 1wIxTkHM_pUkFqOnAAAA with id 57770281
Chrome 63.0.3239 (Mac OS X 10.12.6): Executed 602 of 1075 SUCCESS (0 secs / 21.871 secs)
Chrome 63.0.3239 (Mac OS X 10.12.6): Executed 611 of 1075 SUCCESS (0 secs / 21.973 secs)
Chrome 63.0.3239 (Mac OS X 10.12.6): Executed 844 of 1075 SUCCESS (0 secs / 23.781 secs)
Chrome 63.0.3239 (Mac OS X 10.12.6): Executed 1075 of 1075 SUCCESS (25.253 secs / 24.916 secs)
Interesting. Are you using sample test source code that you can post somewhere?
I also would have expected the 1,075 tests (123 matches of describe() across 122 files) to have been divided by 4 and then executed across the browsers.
That is exactly the intention of this project. From the log output, it looks like its running all of the tests, 4 times instead of splitting them up. Is that correct? If so, that would be a bug.
I wonder if there's a cool/fast way to help speed up the loading of 4 browsers. Maybe one browser, then 4 tabs within it?
I have found using ChromeHeadless has a faster startup time, likely to the fact that it does not need to initialize the visual rendering part of the browser. I hadn't thought of attempting multiple tabs... that would be cool. I guess there might be a tradeoff of possible conflicting with other parallel running test suite by manipulating shared storage for the single domain (local-storage, cookies, etc...). Would be a cool feature to toggle though. Do you know of any other karma plugins that uses tabs?
Do you have any update on this? I have not been able to reproduce your results. My only thought is that you are using an old version of karma (about 2yrs) that my plugin may not be compatible with. I have not tested that far back. Try upgrading and see if it fixes it
Closing due to inactivity.
Came across this and got these results running on a MBP 15" Early 2013 (8 cores supposedly):
1x --------------
real 1m25.659s
user 1m42.960s
sys 0m10.470s
2x --------------
real 1m15.996s
user 2m1.312s
sys 0m13.601s
3x --------------
real 1m11.746s
user 2m23.417s
sys 0m18.218s
4x --------------
real 1m13.970s
user 2m36.800s
sys 0m19.698s
7x (default as cpu-cores -1):
31 05 2018 09:54:53.081:WARN [launcher]: Chrome was not killed in 2000 ms, sending SIGKILL.
...
Output looks like the following. Dunno if it helps:
$ time ng test --watch false
31 05 2018 09:50:12.415:INFO [framework:karma-parallel]: sharding specs across 4 browsers
10% building modules 1/1 modules 0 active31 05 2018 09:50:24.086:INFO [karma]: Karma v2.0.0 server started at http://0.0.0.0:9876/
31 05 2018 09:50:24.086:INFO [launcher]: Launching browsers ChromeCustom, ChromeCustom, ChromeCustom, ChromeCustom with unlimited concurrency
31 05 2018 09:50:24.092:INFO [launcher]: Starting browser Chrome
31 05 2018 09:50:24.101:INFO [launcher]: Starting browser Chrome
31 05 2018 09:50:24.105:INFO [launcher]: Starting browser Chrome
31 05 2018 09:50:24.110:INFO [launcher]: Starting browser Chrome
31 05 2018 09:50:53.683:INFO [Chrome 66.0.3359 (Mac OS X 10.13.4)]: Connected on socket 9BZR5t24QtttuU6JAAAA with id 28541602
31 05 2018 09:50:53.686:INFO [Chrome 66.0.3359 (Mac OS X 10.13.4)]: Connected on socket 2rQGBDF22x_k02bmAAAB with id 98195268
31 05 2018 09:50:53.688:INFO [Chrome 66.0.3359 (Mac OS X 10.13.4)]: Connected on socket wq0Cgfjrl0OHOBVlAAAC with id 25037401
31 05 2018 09:50:53.702:INFO [Chrome 66.0.3359 (Mac OS X 10.13.4)]: Connected on socket Z7q7CTd-qtBqLDm8AAAD with id 48113952
...
Chrome 66.0.3359 (Mac OS X 10.13.4): Executed 227 of 227 SUCCESS (10.476 secs / 0 secs)
Chrome 66.0.3359 (Mac OS X 10.13.4): Executed 170 of 170 SUCCESS (12.899 secs / 0 secs)
Chrome 66.0.3359 (Mac OS X 10.13.4): Executed 196 of 196 SUCCESS (11.819 secs / 0 secs)
Chrome 66.0.3359 (Mac OS X 10.13.4): Executed 245 of 245 SUCCESS (19.952 secs / 0 secs)
TOTAL: 838 SUCCESS
TOTAL: 838 SUCCESS
TOTAL: 838 SUCCESS
real 1m13.970s
user 2m36.800s
sys 0m19.698s
I'm also not really seeing any benefit. I think about 5-10 seconds were shaved but was kind of expecting at least half the time from x2 :)
227 + 170 + 196 + 245 = 838 so maybe it works but doesn't seem faster with Chrome or ChromeHeadless
@joeljeske What kind of results do you get?
@joeljeske Can you give me some insight on how to tell how long each test/test suite takes? That will help me help you.
I know this issue is closed. But rather than opening another (which isn't really an issue anyways) I add this here instead.
Like @intellix mentioned, I'm seeing very little benefit and in some cases even slower test execution. My project is an Angular CLI project with 1000+ tests.
Was expecting a decent performance boost but maybe the setup of each test suite takes up more time only running on one core.
Was wondering if anyone has worked with a Angular CLI project and managed to work around this issue?