Executors and test maximums Question

[D. Turco]

• **I'm submitting a ... **

  • bug report
  • feature request
  • question

• Do you want to request a feature or report a bug?

• What is the current behavior?
  We currently have 1600+ mocha tests. I noticed when running the default executor pattern cores - 1 with chrome headless browser, not all tests get ran. Its only when i bump the executor count to 6 that i get the count over 1600, but then i do also notice this plugin seems to add an additional 15-20 tests to the run count, which i can only assume is from it having to keep the browser alive or something.

My question is: Is there a relationship between what the browser can handle and far as test count limits and the executor pattern needed? Does it warrant a new executor pattern that can figure total test count/batch to make it all work better. I dont love the thought that i would have to keep track of tests counts and go update the executor count to ensure they all can run. This i would think will eventually not work anyway, 3000+ tests means what 10+ executors-ish not even sure at that point what server requirements would be needed to account for that high of a test count..

This plugin was used because karma-mocha already seems to hit the limit of tests ran and we get capped around 950 tests that ever run without this plugin..

Any insight or guidance would be great..

• version:

"karma": "3.1.4",
"karma-chrome-launcher": "2.2.0",
"karma-coverage": "1.1.2",
"karma-coverage-istanbul-reporter": "2.0.4",
"karma-es6-shim": "1.0.0",
"karma-intl-shim": "1.0.3",
"karma-mocha": "1.3.0",
"karma-mocha-reporter": "2.2.5",
"karma-parallel": "0.3.1",
"karma-sinon-chai": "2.0.2",
"karma-sourcemap-loader": "0.3.7",
"karma-spec-reporter": "0.0.32",
"karma-webpack": "3.0.5",

• Browser:
  Google Chrome is up to date
  Version 71.0.3578.98 (Official Build) (64-bit)

ChromeHeadless: {
base: 'Chrome',
flags: [
'--headless',
'--disable-gpu',
'--log-level=3',
'--disable-web-security',
'--disable-search-geolocation-disclosure',
// Without a remote debugging port, Google Chrome exits immediately.
'--remote-debugging-port=9222',
],
}

• Language: Typescript

[Joel Jeske]

Hmm there is no upper limit on the number of tests. It does depend on how expensive each test is in terms of CPU or memory.

It is true that karma parallel adds some tests in some circumstances, but that is only when it detects that there are either not enough specs for each browser, or you are focusing a test in browser and therefore it needs to create a fake focused test in another browser to prevent all tests from running.

Your question concerns me, though. Why were you having trouble running all tests with karma mocha? It shouldn’t care how many tests you have. Karma parallel should speed things up, but not support more tests. I suspect you have some other issue that prevents all tests from running properly, not sure what that might though.

I do not suspect any issue with the strategy pattern, but if you wanted to experiment you can use a custom strategy. Look at the readme for info.

[D. Turco]

Thanks for response. here is a sample output from the parallel runner. Its seems odd the highlighted one is not 143/143 meaning all ran like the rest. seems like its skipping tests all together.

HeadlessChrome 72.0.3626 (Windows 8.1.0.0): Executed 200 of 200�[32m SUCCESS�[39m (11.557 secs / 8.1 secs)
HeadlessChrome 72.0.3626 (Windows 8.1.0.0): Executed 273 of 273�[32m SUCCESS�[39m (19.781 secs / 14.645 secs)
HeadlessChrome 72.0.3626 (Windows 8.1.0.0): Executed 208 of 208�[32m SUCCESS�[39m (8.559 secs / 4.681 secs)
HeadlessChrome 72.0.3626 (Windows 8.1.0.0): Executed 201 of 201�[32m SUCCESS�[39m (16.337 secs / 11.789 secs)
**

HeadlessChrome 72.0.3626 (Windows 8.1.0.0): Executed 104 of 143�[32m SUCCESS�[39m (7.131 secs / 4.808 secs)

**
HeadlessChrome 72.0.3626 (Windows 8.1.0.0): Executed 187 of 187�[32m SUCCESS�[39m (6.432 secs / 3.435 secs)
HeadlessChrome 72.0.3626 (Windows 8.1.0.0): Executed 267 of 267�[32m SUCCESS�[39m (15.5 secs / 10.738 secs)
HeadlessChrome 72.0.3626 (Windows 8.1.0.0): Executed 147 of 147�[32m SUCCESS�[39m (3.489 secs / 1.382 secs)
HeadlessChrome 72.0.3626 (Windows 8.1.0.0): Executed 210 of 210�[32m SUCCESS�[39m (18.05 secs / 13.587 secs)
[32mTOTAL: 1797 SUCCESS�[39m]

[D. Turco]

it turns out vuex and inversify was causing the issues.. we had to refactor all of our tests and hijack resource resolutions with inversify middleware.. now all tests run and parallel def speeds it all up..