Tests are actually running in all browser instances
opened this issue · comments
-
**I'm submitting a ... **
- [ X] bug report
- feature request
- support request => Please do not submit support request here, see note at the top of this template.
-
Do you want to request a feature or report a bug?
Bug -
What is the current behavior?
I'm getting all tests running on every browser instance. -
If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem via
- set karma config like:
{
plugins: ['karma-chrome-launcher', 'karma-coverage', 'karma-junit-reporter', 'karma-mocha', 'karma-parallel'
],
basePath: './',
frameworks: ['parallel', 'mocha'],
parallelOptions: {
shardStrategy: 'round-robin',
aggregatedReporterTest: /coverage|istanbul|junit/i
},
files: [
// stripped for brevity
'../test/**/*.spec.js'
],
preprocessors: {
'scripts/**/*.js': 'coverage'
},
proxies: {
'/apps': 'http://localhost:9000/apps',
'/images': 'http://localhost:8080/images',
'/resources': 'http://localhost:8080/resources',
'/foo': 'http://localhost:8080/foo'
},
exclude: [],
reporters: ['progress', 'coverage', 'junit'],
junitReporter: {
outputFile: 'test-results.xml',
suite: ''
},
coverageReporter: {
type: 'cobertura',
dir: 'coverage/'
},
port: 9876,
runnerPort: 9100,
colors: true,
logLevel: config.LOG_INFO,
autoWatch: true,
browsers: ['HeadlessChrome'],
customLaunchers: {
'HeadlessChrome': {
base: 'ChromeHeadless',
flags: [
'--no-sandbox'
]
}
},
browserNoActivityTimeout: 120000,
captureTimeout: 120000,
singleRun: false
}
- Setup an desribe.only()
- Within the only, create several it()s that
throws "whatever1"... to make them throw when the tests run - run the tests
When you look at the junit report, it has 3 (cpus-1) instances in it:
<testcase name="Utils | #isDateValid() | should return FALSE when the date is before today or it is invalid |" time="0" classname="HeadlessChrome_65_0_3325_(Mac_OS_X_10_13_4).Utils |">
<failure type="">Error: the string "crap1" was thrown, throw an Error :)
</failure>
</testcase>
<testcase name="Utils | #isDateValid() | should return FALSE when the date is before today or it is invalid |" time="0.001" classname="HeadlessChrome_65_0_3325_(Mac_OS_X_10_13_4).Utils |">
<failure type="">Error: the string "crap1" was thrown, throw an Error :)
</failure>
</testcase>
<testcase name="Utils | #isDateValid() | should return FALSE when the date is before today or it is invalid |" time="0" classname="HeadlessChrome_65_0_3325_(Mac_OS_X_10_13_4).Utils |">
<failure type="">Error: the string "crap1" was thrown, throw an Error :)
</failure>
</testcase>
Also, the console shows 3:
HeadlessChrome 65.0.3325 (Mac OS X 10.13.4) Utils | #isDateValid() | should return FALSE when the date is before today or it is invalid | FAILED
[exec] Error: the string "crap1" was thrown, throw an Error :)
HeadlessChrome 65.0.3325 (Mac OS X 10.13.4) Utils | #isDateValid() | should return FALSE when the date is before today or it is invalid | FAILED
[exec] Error: the string "crap1" was thrown, throw an Error :)
HeadlessChrome 65.0.3325 (Mac OS X 10.13.4) Utils | #isDateValid() | should return FALSE when the date is before today or it is invalid | FAILED
[exec] Error: the string "crap1" was thrown, throw an Error :)
-
What is the expected behavior?
I'm actually not sure how this would look when it works, but I would assume that a test would get skipped in "other" browsers, and, and I don't see any tests getting skipped. -
What is the motivation / use case for changing the behavior?
The current behavior makes the plugin worthless. (I hope there is something weird about my settings or versions being used, as I can't imagine this is how it works for everyone else?) -
Please tell us about your environment:
- version: 0.2.4
- Browser: ChromeHeadless 65.0.3325
- Language: all
- package.json contains these dependencies:
"chai": "1.10.0",
"gulp": "3.9.1",
"gulp-util": "3.0.8",
"istanbul": "0.4.5",
"karma": "2.0.0",
"karma-chai": "0.1.0",
"karma-chrome-launcher": "2.2.0",
"karma-coverage": "1.1.1",
"karma-junit-reporter": "1.2.0",
"karma-mocha": "1.3.0",
"karma-parallel": "0.2.4",
"mocha": "2.3.2",
"sinon": "1.10.2",
"sinon-chai": "3.0.0"
-
Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. stackoverflow, gitter, etc)
For example, when running 4 executors, with 3297 tests, the progress reporter looks like this:
[exec] HeadlessChrome 65.0.3325 (Mac OS X 10.13.4): Executed 1225 of 3297 (3 FAILED) (0 secs / 19.645 secs)
[exec] HeadlessChrome 65.0.3325 (Mac OS X 10.13.4): Executed 1222 of 3297 (3 FAILED) (0 secs / 17.717 secs)
[exec] HeadlessChrome 65.0.3325 (Mac OS X 10.13.4): Executed 1224 of 3297 (3 FAILED) (0 secs / 19.068 secs)
[exec] HeadlessChrome 65.0.3325 (Mac OS X 10.13.4): Executed 1225 of 3297 (3 FAILED) (0 secs / 17.414 secs)
Interesting. Your expectations of this plugin is correct. I am not getting that behavior. Can you create a simple project that reproduces this?
Also, you might try it without the '--no-sandbox' custom browser flag. This plugin has to distinguish between each browser instance and it does so by using a cookie on initial load. If they are not sandboxed, it might cause a race condition in which 2 or more different browser instances, may operate on the same set of test files. (I have tried turning on --no-sandbox in my test project but am not getting incorrect results but it may still be worth trying)
@Matt812 Did you ever try to remove the --no-sandbox flag? I'm just curious if that is an issue that should be mentioned in the readme.
I ran into the same issue. However, I was using PhantomJS and switching to ChromeHeadless resolved it for me. Obviously, mine had a different cause than the OP, but I figured I should post this here in case someone else comes along with the same problem.
Closing due to inactivity.
Have the same problem with karma-parallel and mocha. They seems to run the all the test in all of the browsers, I debuged it and the ctx.it and so on, doesn't override it of the mocha ones.
This is the result of karma-parallel with mocha:
12:56:35.597 HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 711 of 729 (skipped 18) SUCCESS (1 min 57.424 secs / 2.677 secs)
12:56:35.597 �[1A�[2K�[1A�[2K�[1A�[2KHeadlessChrome 0.0.0 (Linux 0.0.0): Executed 711 of 729 (skipped 18) SUCCESS (1 min 57.139 secs / 2.898 secs)
12:56:35.597 HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 711 of 729 (skipped 18) SUCCESS (0 secs / 2.799 secs)
12:56:35.597 HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 711 of 729 (skipped 18) SUCCESS (1 min 57.424 secs / 2.677 secs)
12:56:35.799 �[1A�[2K�[1A�[2K�[1A�[2KHeadlessChrome 0.0.0 (Linux 0.0.0): Executed 711 of 729 (skipped 18) SUCCESS (1 min 57.139 secs / 2.898 secs)
12:56:35.799 HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 711 of 729 (skipped 18) SUCCESS (1 min 57.747 secs / 2.799 secs)
12:56:35.799 HeadlessChrome 0.0.0 (Linux 0.0.0): Executed 711 of 729 (skipped 18) SUCCESS (1 min 57.424 secs / 2.677 secs)
12:56:36.101 TOTAL: 2133 SUCCESS
12:56:36.101
12:56:36.101 Finished in 5 mins 52.31 secs / 8.374 secs @ 12:56:36 GMT+0200 (CEST)
12:56:36.101
12:56:36.101 SUMMARY:
12:56:36.101 ✔ 2133 tests completed
12:56:36.101 ℹ 54 tests skipped
12:56:36.101 TOTAL: 2133 SUCCESS
and this is what I get from running without karma-parallel:
16:27:07.184 �[1A�[2KHeadlessChrome 0.0.0 (Linux 0.0.0): Executed 711 of 729 (ski SUCCESS (1 min 22.59 secs / 2.849 secs)
16:27:07.185 TOTAL: 711 SUCCESS
16:27:07.185 TOTAL: 711 SUCCESS
16:27:08.89
16:27:08.89 Finished in 1 min 22.59 secs / 2.849 secs @ 16:27:07 GMT+0200 (CEST)
16:27:08.89
16:27:08.89 SUMMARY:
16:27:08.89 ✔ 711 tests completed
16:27:08.89 ℹ 18 tests skipped
@sky007 I am facing the same issue ? Did you find any resolution for the above ?
@joeljeske Can we re-open this issue ? I tried without sandbox flag and still the issue