Merging with nyc storybook test coverage with other reports generated by coverageProvider 'istanbul' result in wrong coverage
meanstackmax opened this issue · comments
Link to bug demonstration repository https://github.com/meanstackmax/demo-test-coverage/tree/main
Expected Behavior
Vitest and storybook should use the same source map outputted, thus the issue with different columns for the same stmt not affecting merged report
Observed Behavior
I am trying to generate a merged code report. Therefore, I am using for my unit tests vitest with istanbul. For Storybook I am using the addon which also uses istanbul. To merge the reports I am using nyc.
The issue I am having is that the reports of vitest and storybook are showing me different columns for the same stmt. This leads to not correct merging.
For example in one component vitest said that there is a stmt on line 38 start at col 18 and ends on the same line.
"2": {
"start": {
"line": 38,
"column": 18
},
"end": {
"line": 38,
"column": null
}
},
However, storybook coverage report says the statement is on line 38 in column 2:
"3": {
"start": {
"line": 38,
"column": 2
},
"end": {
"line": 38,
"column": null
}
},
Troubleshooting steps
-Tried to add this to package.json, but it didn't work:
"overrides": {
"istanbul-lib-instrument": "6.0.1",
"istanbul-lib-coverage": "3.2.0"
},
- Tried to upgrade packages to "@storybook/addon-coverage" to v 1.0.1 and "@storybook/test-runner" to v 0.17.0 but it doesn't resolve the issue
- Tried manually enabling source map for storybook and vitest and adding nyc configuration (installing "@istanbuljs/nyc-config-typescript" and adding .nycrc file for more control), but it didn't work as well
- Tried with storybook v8 and storybook the dependancies updated but it doesn't solve the issue as well
- Tried using istanbul-merge instead of nyc for merging the reports, but in the end it basically ends up with the same output as using nyc
Environment Information
System:
OS: macOS 14.4.1
CPU: (8) arm64 Apple M1
Memory: 122.00 MB / 16.00 GB
Binaries:
Node: 18.16.1 - ~/.nvm/versions/node/v18.16.1/bin/node
Yarn: 1.22.19 - ~/.yarn/bin/yarn
npm: 9.5.1 - ~/.nvm/versions/node/v18.16.1/bin/npm
npmPackages:
@babel/core: ^7.23.7 => 7.24.5
@vitest/coverage-istanbul: ^1.2.1 => 1.5.3
babel-loader: ^9.1.3 => 9.1.3
nyc: ^15.1.0 => 15.1.0
typescript: ^5.3.3 => 5.4.5
There are 3 different versions of istanbul-lib-instrument
in use after running npm list istanbul-lib-instrument
, it should be the cause of the problem.
coverage-demo-app@1.0.0 /demo-test-coverage
├─┬ @storybook/addon-coverage@1.0.1
│ ├─┬ @jsdevtools/coverage-istanbul-loader@3.0.5
│ │ └── istanbul-lib-instrument@4.0.3
│ ├── istanbul-lib-instrument@6.0.2
│ └─┬ vite-plugin-istanbul@3.0.4
│ └── istanbul-lib-instrument@5.2.1
├─┬ @storybook/addon-essentials@7.6.18
│ └─┬ @storybook/addon-docs@7.6.18
│ └─┬ @jest/transform@29.7.0
│ └─┬ babel-plugin-istanbul@6.1.1
│ └── istanbul-lib-instrument@5.2.1
├─┬ @vitest/coverage-istanbul@1.5.3
│ └── istanbul-lib-instrument@6.0.2 deduped
├─┬ jest@29.7.0
│ └─┬ @jest/core@29.7.0
│ └─┬ @jest/reporters@29.7.0
│ └── istanbul-lib-instrument@6.0.2 deduped
└─┬ nyc@15.1.0
└── istanbul-lib-instrument@4.0.3
@cenfun thank you, after syncing all istanbul-lib-instrument
it seems like an issue still present. It looks like it doesn't count on storybook tests. So I did the following:
- Added the following overrides section to my
package.json
and ran$npx npm-force-resolutions
:
"overrides": {
"@storybook/addon-coverage": {
"istanbul-lib-instrument": "^6.0.2"
},
"@jsdevtools/coverage-istanbul-loader": {
"istanbul-lib-instrument": "^6.0.2"
},
"vite-plugin-istanbul": {
"istanbul-lib-instrument": "^6.0.2"
},
"@jest/reporters": {
"istanbul-lib-instrument": "^6.0.2"
},
"@vitest/coverage-istanbul": {
"istanbul-lib-instrument": "^6.0.2"
},
"nyc": {
"istanbul-lib-instrument": "^6.0.2"
},
"babel-plugin-istanbul": {
"istanbul-lib-instrument": "^6.0.2"
}
},
Even after adding specs and storybook the coverage is 95.45% (expected: 100%). The original idea was to have smoke test coverage report