joeljeske / karma-parallel

A Karma JS Framework to support sharding tests to run in parallel across multiple browsers

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Tests are actually running in all browser instances

opened this issue · comments

  • **I'm submitting a ... **

    • [ X] bug report
    • feature request
    • support request => Please do not submit support request here, see note at the top of this template.
  • Do you want to request a feature or report a bug?
    Bug

  • What is the current behavior?
    I'm getting all tests running on every browser instance.

  • If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem via

    • set karma config like:
    {
        plugins: ['karma-chrome-launcher', 'karma-coverage', 'karma-junit-reporter', 'karma-mocha', 'karma-parallel'
        ],
        basePath: './',
        frameworks: ['parallel', 'mocha'],
        parallelOptions: {
            shardStrategy: 'round-robin',
            aggregatedReporterTest: /coverage|istanbul|junit/i
        },
        files: [
// stripped for brevity
            '../test/**/*.spec.js'
        ],
        preprocessors: {
            'scripts/**/*.js': 'coverage'
        },
        proxies: {
            '/apps': 'http://localhost:9000/apps',
            '/images': 'http://localhost:8080/images',
            '/resources': 'http://localhost:8080/resources',
            '/foo': 'http://localhost:8080/foo'
        },
        exclude: [],
        reporters: ['progress', 'coverage', 'junit'],
        junitReporter: {
            outputFile: 'test-results.xml',
            suite: ''
        },
        coverageReporter: {
            type: 'cobertura',
            dir: 'coverage/'
        },
        port: 9876,
        runnerPort: 9100,
        colors: true,
        logLevel: config.LOG_INFO,
        autoWatch: true,
        browsers: ['HeadlessChrome'],
        customLaunchers: {
            'HeadlessChrome': {
                base: 'ChromeHeadless',
                flags: [
                    '--no-sandbox'
                ]
            }
        },
        browserNoActivityTimeout: 120000,
        captureTimeout: 120000,
        singleRun: false
    }
  • Setup an desribe.only()
  • Within the only, create several it()s that throws "whatever1" ... to make them throw when the tests run
  • run the tests

When you look at the junit report, it has 3 (cpus-1) instances in it:

  <testcase name="Utils | #isDateValid() | should return FALSE when the date is before today or it is invalid |" time="0" classname="HeadlessChrome_65_0_3325_(Mac_OS_X_10_13_4).Utils |">
    <failure type="">Error: the string "crap1" was thrown, throw an Error :)
</failure>
  </testcase>
  <testcase name="Utils | #isDateValid() | should return FALSE when the date is before today or it is invalid |" time="0.001" classname="HeadlessChrome_65_0_3325_(Mac_OS_X_10_13_4).Utils |">
    <failure type="">Error: the string "crap1" was thrown, throw an Error :)
</failure>
  </testcase>
  <testcase name="Utils | #isDateValid() | should return FALSE when the date is before today or it is invalid |" time="0" classname="HeadlessChrome_65_0_3325_(Mac_OS_X_10_13_4).Utils |">
    <failure type="">Error: the string "crap1" was thrown, throw an Error :)
</failure>
  </testcase>

Also, the console shows 3:

            HeadlessChrome 65.0.3325 (Mac OS X 10.13.4) Utils | #isDateValid() | should return FALSE when the date is before today or it is invalid | FAILED
     [exec]     Error: the string "crap1" was thrown, throw an Error :)
            HeadlessChrome 65.0.3325 (Mac OS X 10.13.4) Utils | #isDateValid() | should return FALSE when the date is before today or it is invalid | FAILED
     [exec]     Error: the string "crap1" was thrown, throw an Error :)
            HeadlessChrome 65.0.3325 (Mac OS X 10.13.4) Utils | #isDateValid() | should return FALSE when the date is before today or it is invalid | FAILED
     [exec]     Error: the string "crap1" was thrown, throw an Error :)
  • What is the expected behavior?
    I'm actually not sure how this would look when it works, but I would assume that a test would get skipped in "other" browsers, and, and I don't see any tests getting skipped.

  • What is the motivation / use case for changing the behavior?
    The current behavior makes the plugin worthless. (I hope there is something weird about my settings or versions being used, as I can't imagine this is how it works for everyone else?)

  • Please tell us about your environment:

    • version: 0.2.4
    • Browser: ChromeHeadless 65.0.3325
    • Language: all
    • package.json contains these dependencies:
      "chai": "1.10.0",
      "gulp": "3.9.1",
      "gulp-util": "3.0.8",
      "istanbul": "0.4.5",
      "karma": "2.0.0",
      "karma-chai": "0.1.0",
      "karma-chrome-launcher": "2.2.0",
      "karma-coverage": "1.1.1",
      "karma-junit-reporter": "1.2.0",
      "karma-mocha": "1.3.0",
      "karma-parallel": "0.2.4",
      "mocha": "2.3.2",
      "sinon": "1.10.2",
      "sinon-chai": "3.0.0"
  • Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. stackoverflow, gitter, etc)

For example, when running 4 executors, with 3297 tests, the progress reporter looks like this:

     [exec] HeadlessChrome 65.0.3325 (Mac OS X 10.13.4): Executed 1225 of 3297 (3 FAILED) (0 secs / 19.645 secs)
     [exec] HeadlessChrome 65.0.3325 (Mac OS X 10.13.4): Executed 1222 of 3297 (3 FAILED) (0 secs / 17.717 secs)
     [exec] HeadlessChrome 65.0.3325 (Mac OS X 10.13.4): Executed 1224 of 3297 (3 FAILED) (0 secs / 19.068 secs)
     [exec] HeadlessChrome 65.0.3325 (Mac OS X 10.13.4): Executed 1225 of 3297 (3 FAILED) (0 secs / 17.414 secs)

Interesting. Your expectations of this plugin is correct. I am not getting that behavior. Can you create a simple project that reproduces this?

Also, you might try it without the '--no-sandbox' custom browser flag. This plugin has to distinguish between each browser instance and it does so by using a cookie on initial load. If they are not sandboxed, it might cause a race condition in which 2 or more different browser instances, may operate on the same set of test files. (I have tried turning on --no-sandbox in my test project but am not getting incorrect results but it may still be worth trying)

@Matt812 Did you ever try to remove the --no-sandbox flag? I'm just curious if that is an issue that should be mentioned in the readme.

I ran into the same issue. However, I was using PhantomJS and switching to ChromeHeadless resolved it for me. Obviously, mine had a different cause than the OP, but I figured I should post this here in case someone else comes along with the same problem.

Closing due to inactivity.

Have the same problem with karma-parallel and mocha. They seems to run the all the test in all of the browsers, I debuged it and the ctx.it and so on, doesn't override it of the mocha ones.

This is the result of karma-parallel with mocha:

12:56:35.597 HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 711 of 729 (skipped 18) SUCCESS (1 min 57.424 secs / 2.677 secs)
12:56:35.597 �[1A�[2K�[1A�[2K�[1A�[2KHeadlessChrome 0.0.0 (Linux 0.0.0): Executed 711 of 729 (skipped 18) SUCCESS (1 min 57.139 secs / 2.898 secs)
12:56:35.597 HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 711 of 729 (skipped 18) SUCCESS (0 secs / 2.799 secs)
12:56:35.597 HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 711 of 729 (skipped 18) SUCCESS (1 min 57.424 secs / 2.677 secs)
12:56:35.799 �[1A�[2K�[1A�[2K�[1A�[2KHeadlessChrome 0.0.0 (Linux 0.0.0): Executed 711 of 729 (skipped 18) SUCCESS (1 min 57.139 secs / 2.898 secs)
12:56:35.799 HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 711 of 729 (skipped 18) SUCCESS (1 min 57.747 secs / 2.799 secs)
12:56:35.799 HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 711 of 729 (skipped 18) SUCCESS (1 min 57.424 secs / 2.677 secs)
12:56:36.101 TOTAL: 2133 SUCCESS
12:56:36.101
12:56:36.101 Finished in 5 mins 52.31 secs / 8.374 secs @ 12:56:36 GMT+0200 (CEST)
12:56:36.101
12:56:36.101 SUMMARY:
12:56:36.101 2133 tests completed
12:56:36.101 54 tests skipped
12:56:36.101 TOTAL: 2133 SUCCESS

and this is what I get from running without karma-parallel:

16:27:07.184 �[1A�[2KHeadlessChrome 0.0.0 (Linux 0.0.0): Executed 711 of 729 (ski SUCCESS (1 min 22.59 secs / 2.849 secs)
16:27:07.185 TOTAL: 711 SUCCESS
16:27:07.185 TOTAL: 711 SUCCESS
16:27:08.89
16:27:08.89 Finished in 1 min 22.59 secs / 2.849 secs @ 16:27:07 GMT+0200 (CEST)
16:27:08.89
16:27:08.89 SUMMARY:
16:27:08.89 711 tests completed
16:27:08.89 18 tests skipped

@sky007 I am facing the same issue ? Did you find any resolution for the above ?
@joeljeske Can we re-open this issue ? I tried without sandbox flag and still the issue