Sharding running same test multiple times in a single run

[Levi Wilson]

• **I'm submitting a ... **

  • bug report
  • feature request

• Do you want to request a feature or report a bug?
  Feature.

• What is the current behavior?
  I ❤️ karma-parallel; though when I run a focused spec (with an fdescribe or the like), it takes longer than if it were to run just across a single executor because it runs tests more than one time.

• If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem
  I ran an fdescribe in jasmine that ultimately had 4 specs. Rather than running just the 4 (I think I have 7 executors based on my cores) it ran 10 specs. Perhaps a rounding in the shard strategy?

HeadlessChrome 0.0.0 (Mac OS X 10.13.6): Executed 1 of 239 (skipped 238) SUCCESS (0.536 secs / 0.012 secs)
HeadlessChrome 0.0.0 (Mac OS X 10.13.6): Executed 1 of 555 (skipped 554) SUCCESS (1.154 secs / 0.013 secs)
HeadlessChrome 0.0.0 (Mac OS X 10.13.6): Executed 1 of 393 (skipped 392) SUCCESS (0.747 secs / 0.013 secs)
HeadlessChrome 0.0.0 (Mac OS X 10.13.6): Executed 1 of 372 (skipped 371) SUCCESS (0.804 secs / 0.012 secs)
HeadlessChrome 0.0.0 (Mac OS X 10.13.6): Executed 4 of 350 (skipped 346) SUCCESS (0.871 secs / 0.207 secs)
HeadlessChrome 0.0.0 (Mac OS X 10.13.6): Executed 1 of 387 (skipped 386) SUCCESS (0.785 secs / 0.014 secs)
HeadlessChrome 0.0.0 (Mac OS X 10.13.6): Executed 1 of 683 (skipped 682) SUCCESS (1.396 secs / 0.009 secs)
TOTAL: 10 SUCCESS

• What is the expected behavior?
  One of two things; either the superfluous spec on the other 6 executors shouldn't run anything, or I'd expect that it spread it out over 4 executors and the remaining 3 would be idle.

Either that or it'd be nice to set a threshold in the configuration where it wouldn't parallelize the specs at all. For example:

parallelOptions: {
  shardThreshold: 50 /* specify anything 50 and below wouldn't parallelize at all
}

• Please tell us about your environment:

• version: 0.2.9
• Browser: all
• Language: all

• Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. stackoverflow, gitter, etc)

[Joel Jeske]

Thank you for the report. As mentioned in the readme, karma-parallel may add additional tests to your output.

If this plugin discovers that you have focused some tests (fit, it.only, etc...) in other browser instances, it will add an extra focused test in the current browser instance to limit the running of the tests in the given browser. Similarly, when dividing up the tests, if there are not enough tests for a given browser, it will add an extra test to prevent karma from failing due to no running tests

Essentially, by the time we know that there is a focused describe block, it is too late to decide not to shard; we have already started up all the browsers instances and started parsing the tests. The only fail-safe way (that I know of) to tell mocha/jasmine to skip other tests, is by adding a focused test. Otherwise, shards that happen to not be delegated any focused tests will attempt to run all specs since it is unaware of any focused specs at all.

The spec that is running, is a very lightweight spec that does nothing at all. See:
https://github.com/joeljeske/karma-parallel/blob/v0.2.9/lib/karma-parallelizer.js#L142

[Levi Wilson]

Makes sense, thanks!

[kaashan]

@joeljeske : we have a project set-up where we use karma-parallel with 5 executors to run ~8500 tests. Below is a snapshot of a successful run:

We have also set jasmine's failSpecWithNoExpectations to true in karma.conf.js file

client: {
      jasmine: {
        random: false,
        failSpecWithNoExpectations: true
      }
},

Recently, for one odd build on our CI/CD pipeline karma-parallel added the lightweight spec you mentioned above and since this spec does not has any expectation, it caused that build to fail:

I can understand additional tests introduced by karma-parallel in case of focused tests or not enough tests to be run for every executor instance. But why did this happen for our use case?

From the failure screenshot, it could be seen that 6956 tests were split up amongst shard-2, shard-3, shard-4 and shard-5. The left 1577 remaining tests should have been received by shard-1 and there was no need for karma-parallel to add any default test. So, what could be the reason for addition of default test in this particular case?