Should fail the test cases split to a browser that is disconnected

[Paul Lan]

• **I'm submitting a ... **

  • bug report
  • feature request

• Do you want to request a feature or report a bug?

bug

• What is the current behavior?

Today my ci-cd pipeline failed unexpectedly after merging a branch into master. It shouldn't have as the code run by karma in that branch are identical before and after merging to master.

I looked into the stdout, and found this error:

11 04 2019 23:34:59.265:WARN [HeadlessChrome 70.0.3538 (Linux 0.0.0)]: Disconnected (0 times)reconnect failed before timeout of 60000ms (ping timeout)
HeadlessChrome 70.0.3538 (Linux 0.0.0) ERROR

To surprise me, karma-parallel didn't count it as a failure.

And after that, my TOTAL test cases was altered mysteriously.

TOTAL: 282 SUCCESS

HeadlessChrome 70.0.3538 (Linux 0.0.0): Executed 140 of 140 SUCCESS (3 mins 16.315 secs / 3 mins 18.967 secs)
HeadlessChrome 70.0.3538 (Linux 0.0.0): Executed 1 of 1 SUCCESS (0.002 secs / 0.002 secs)
HeadlessChrome 70.0.3538 (Linux 0.0.0): Executed 141 of 141 SUCCESS (5 mins 7.221 secs / 5 mins 9.639 secs)
TOTAL: 282 SUCCESS

I should have more than 450 test cases in total.

• If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

  • Try to create a test case that stop a browser instance with a debugger keyword and open the browser instance's Chrome Debugger so that the browser is effectively disconnected with no response; Or, you may also stop it by creating a very long running time test case (use the third argument to jasmine it function)

• What is the expected behavior?

  • Test cases should fail
  • Total test cases shouldn't change

• Please tell us about your environment:

  • version: 2.0.0-beta.X
  • Browser: [ChromeHeadless 70.0.3538 in Docker with --no-sandbox]
  • Language: [TypeScript 2.6]
  • karma config: browserDisconnectTolerance:5

    frameworks: ['parallel', 'jasmine', '@angular/cli'],
    plugins: [
        require('karma-parallel'),
        require('karma-jasmine'),
        require('karma-chrome-launcher'),
        require('karma-remap-istanbul'),
        require('karma-coverage-istanbul-reporter'),
        require('@angular/cli/plugins/karma'),
        require('karma-spec-reporter'),
    ],
    parallelOptions: {
      executors: 3, // Defaults to cpu-count - 1
      shardStrategy: 'round-robin'
    },
    customLaunchers: {
        ChromeCustom: {
          base: 'ChromeHeadless',
          flags: ['--no-sandbox']
        }
      },
    browsers: ['ChromeCustom'],
    singleRun: true,
    sourcemaps: true,
    logLevel: config.LOG_INFO,
    reporters: ['spec']

• Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. stackoverflow, gitter, etc)

[Wilson Hobbs]

+1

[Volodymyr Kot]

@joeljeske Any thoughts on this one? This makes karma-parallel almost unusable for us because CI becomes unstable. At least feasible workaround would be highly appreciated.

[Nikola Vladimirov Iliev]

Hello guys,
I opened pull request, so you can check the fix in #43.

What I found is that there are 3 issues:

1. Karma 1.6.0 if you are using this version it has issues with browser reconnect functionality related to this article: http://karma-runner.github.io/4.0/about/changelog.html. The fix is available in version 3.1.4.
2. When karma tries to "reconnect" the browser it triggers "browser_error" event first and then "browser_register" event again, which reflects in the shardIndexMap with same browser id but with wrong shard ID. For example lets say you have 3 browsers with ids: 0, 1, 2. And the browser with shard id 2 is disconnected, on reconnect his shard id will become 3, which is wrong. So the fix is to find available shard id.
3. For some reason when the browser is reconnected his shard id is string (should be number) and the check inside the strategy functions is failing, so this is the reason why karma-parallel is adding one fake test in the disconnected browser instance.
   For example: (count % executors) === shardID => (3 % 3) === '0' -> False (Add fake test instead). The fix is to cast them like: Number(a % b) === Number(c).

[Nikola Vladimirov Iliev]

Hello guys,
I opened pull request, so you can check the fix in #43.

What I found is that there are 3 issues:

1. Karma 1.6.0 if you are using this version it has issues with browser reconnect functionality related to this article: http://karma-runner.github.io/4.0/about/changelog.html. The fix is available in version 3.1.4.
2. When karma tries to "reconnect" the browser it triggers "browser_error" event first and then "browser_register" event again, which reflects in the shardIndexMap with same browser id but with wrong shard ID. For example lets say you have 3 browsers with ids: 0, 1, 2. And the browser with shard id 2 is disconnected, on reconnect his shard id will become 3, which is wrong. So the fix is to find available shard id.
3. For some reason when the browser is reconnected his shard id is string (should be number) and the check inside the strategy functions is failing, so this is the reason why karma-parallel is adding one fake test in the disconnected browser instance.
   For example: (count % executors) === shardID => (3 % 3) === '0' -> False (Add fake test instead). The fix is to cast them like: Number(a % b) === Number(c).