Not working with Storybook 7
jfairley opened this issue · comments
Logs and screenshots
click for logs...
[storybook-with-cypress] yarn test
yarn run v1.22.17
$ DEBUG=code-coverage cypress run
[62033:0519/165728.394350:ERROR:node_bindings.cc(279)] Most NODE_OPTIONs are not supported in packaged apps. See documentation for more details.
[62033:0519/165728.394372:ERROR:node_bindings.cc(279)] Most NODE_OPTIONs are not supported in packaged apps. See documentation for more details.
[62033:0519/165728.394376:ERROR:node_bindings.cc(279)] Most NODE_OPTIONs are not supported in packaged apps. See documentation for more details.
DevTools listening on ws://127.0.0.1:54544/devtools/browser/b64f05d0-47fb-4b88-a3c7-35082697cdab
code-coverage combined NYC options { 'report-dir': './coverage', reporter: [ 'lcov', 'clover', 'json', 'json-summary' ], extension: [ '.js', '.cjs', '.mjs', '.ts', '.tsx', '.jsx' ], excludeAfterRemap: false, all: true, include: [ 'stories/**/*' ] } +0ms
====================================================================================================
(Run Starting)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress: 12.12.0 │
│ Browser: Electron 106 (headless) │
│ Node Version: v18.15.0 (/Users/jfairley/n/bin/node) │
│ Specs: 1 found (button.cy.js) │
│ Searched: cypress/e2e/**/*.cy.{js,jsx,ts,tsx} │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────
Running: button.cy.js (1 of 1)
Button
code-coverage NYC file /Users/jfairley/dev/code/storybook-with-cypress/.nyc_output/out.json has 10 key(s) +3s
code-coverage 1 key /Users/jfairley/dev/code/storybook-with-cypress/cypress/support/commands.js file path /Users/jfairley/dev/code/storybook-with-cypress/cypress/support/commands.js +0ms
code-coverage 2 key /Users/jfairley/dev/code/storybook-with-cypress/cypress/support/e2e.js file path /Users/jfairley/dev/code/storybook-with-cypress/cypress/support/e2e.js +0ms
code-coverage 3 key /Users/jfairley/dev/code/storybook-with-cypress/cypress.config.js file path /Users/jfairley/dev/code/storybook-with-cypress/cypress.config.js +0ms
code-coverage in file /Users/jfairley/dev/code/storybook-with-cypress/.nyc_output/out.json all files are not found? false +0ms
code-coverage NYC file /Users/jfairley/dev/code/storybook-with-cypress/.nyc_output/out.json has 10 key(s) +0ms
code-coverage nyc needs to report on all included files +0ms
code-coverage include all files options: { all: true, include: [ 'stories/**/*' ], exclude: undefined, extension: [ '.js', '.cjs', '.mjs', '.ts', '.tsx', '.jsx' ] } +1ms
code-coverage searching files to include using patterns [ 'stories/**/*', '!**/node_modules/**' ] +0ms
code-coverage found 3 file(s) +4ms
/Users/jfairley/dev/code/storybook-with-cypress/stories/Button.jsx
/Users/jfairley/dev/code/storybook-with-cypress/stories/Button.stories.js
/Users/jfairley/dev/code/storybook-with-cypress/stories/button.css
code-coverage coverage has 10 record(s) +1ms
coverage has the following first paths
/Users/jfairley/dev/code/storybook-with-cypress/cypress/support/commands.js
/Users/jfairley/dev/code/storybook-with-cypress/cypress/support/e2e.js
/Users/jfairley/dev/code/storybook-with-cypress/cypress.config.js
/Users/jfairley/dev/code/storybook-with-cypress/stories/Button.stories.js
code-coverage calling NYC reporter with options { 'report-dir': '/Users/jfairley/dev/code/storybook-with-cypress/coverage', reporter: [ 'lcov', 'clover', 'json', 'json-summary' ], extension: [ '.js', '.cjs', '.mjs', '.ts', '.tsx', '.jsx' ], excludeAfterRemap: false, all: true, include: [ 'stories/**/*' ], 'temp-dir': '/Users/jfairley/dev/code/storybook-with-cypress/.nyc_output', tempDir: '/Users/jfairley/dev/code/storybook-with-cypress/.nyc_output', reportDir: '/Users/jfairley/dev/code/storybook-with-cypress/coverage' } +6ms
code-coverage current working directory is /Users/jfairley/dev/code/storybook-with-cypress +0ms
code-coverage after reporting, returning the report folder name /Users/jfairley/dev/code/storybook-with-cypress/coverage +51ms
code-coverage Final coverage in /Users/jfairley/dev/code/storybook-with-cypress/coverage/coverage-final.json +0ms
code-coverage There are 10 key(s) in /Users/jfairley/dev/code/storybook-with-cypress/coverage/coverage-final.json +0ms
code-coverage ❓ /Users/jfairley/dev/code/storybook-with-cypress/cypress.config.js statements covered 0/0 +0ms
code-coverage ❓ /Users/jfairley/dev/code/storybook-with-cypress/coverage/lcov-report/block-navigation.js statements covered 0/0 +0ms
code-coverage ❓ /Users/jfairley/dev/code/storybook-with-cypress/coverage/lcov-report/prettify.js statements covered 0/0 +0ms
code-coverage ❓ /Users/jfairley/dev/code/storybook-with-cypress/coverage/lcov-report/sorter.js statements covered 0/0 +0ms
code-coverage ❓ /Users/jfairley/dev/code/storybook-with-cypress/cypress/e2e/button.cy.js statements covered 0/0 +0ms
code-coverage ❓ /Users/jfairley/dev/code/storybook-with-cypress/cypress/support/commands.js statements covered 0/0 +0ms
code-coverage ❓ /Users/jfairley/dev/code/storybook-with-cypress/cypress/support/e2e.js statements covered 0/0 +0ms
code-coverage ❓ /Users/jfairley/dev/code/storybook-with-cypress/stories/Button.jsx statements covered 0/0 +0ms
code-coverage ❓ /Users/jfairley/dev/code/storybook-with-cypress/stories/Button.stories.js statements covered 0/0 +0ms
code-coverage ❓ /Users/jfairley/dev/code/storybook-with-cypress/stories/button.css statements covered 0/0 +0ms
✓ should render (329ms)
1 passing (1s)
(Results)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Tests: 1 │
│ Passing: 1 │
│ Failing: 0 │
│ Pending: 0 │
│ Skipped: 0 │
│ Screenshots: 0 │
│ Video: true │
│ Duration: 1 second │
│ Spec Ran: button.cy.js │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
(Video)
- Started processing: Compressing to 32 CRF
- Finished processing: 0 seconds
- Video output: /Users/jfairley/dev/code/storybook-with-cypress/cypress/videos/button.cy.js.mp4
====================================================================================================
(Run Finished)
Spec Tests Passing Failing Pending Skipped
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ ✔ button.cy.js 00:01 1 1 - - - │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
✔ All specs passed! 00:01 1 1 - - -
✨ Done in 6.19s.Versions
- What is this plugin's version? If this is NOT the latest released version can you try the latest version, please?
- 3.10.4
- If the plugin worked before in version X, but stopped after upgrading to version Y, please try the released versions between X and Y to see where the breaking change was.
- The break occurs when upgrading storybook to v7.
- What is Cypress version?
- 12.12.0
- What is your operating system?
- MacOS Ventura 13.1
- What is the shell?
- zsh
- What is the Node version?
- 18.15.0
- What is the NPM version?
- 9.5.0
- How do you instrument your application? Cypress does not instrument web application code, so you need to do it yourself.
- I have two methods applied, both of which I suspect would work without the other, but I'm trying to cover my bases.
"@storybook/addon-coverage""babel-plugin-istanbul"registered in.babelrc.json
- I have two methods applied, both of which I suspect would work without the other, but I'm trying to cover my bases.
- When running tests, if you open the web application in regular browser, and open DevTools, do you see
window.__coverage__object? Can you paste a screenshot?- yes
-
- Is there
.nyc_outputfolder? Is there.nyc_output/out.jsonfile. Is it empty? Can you paste at least part of it so we can see the keys and file paths?-
Not fully empty, but no coverage reported in it.
-
see `.nyc_output/out.json`
{ "/Users/jfairley/dev/code/storybook-with-cypress/cypress/support/commands.js": { "path": "/Users/jfairley/dev/code/storybook-with-cypress/cypress/support/commands.js", "statementMap": {}, "fnMap": {}, "branchMap": {}, "s": {}, "f": {}, "b": {}, "_coverageSchema": "1a1c01bbd47fc00a2c39e90264f33305004495a9", "hash": "b100d0ceb86dc20ac064a45298efa0bdb436b8da" }, "/Users/jfairley/dev/code/storybook-with-cypress/cypress/support/e2e.js": { "path": "/Users/jfairley/dev/code/storybook-with-cypress/cypress/support/e2e.js", "statementMap": {}, "fnMap": {}, "branchMap": {}, "s": {}, "f": {}, "b": {}, "_coverageSchema": "1a1c01bbd47fc00a2c39e90264f33305004495a9", "hash": "3c1c42d7c6490def901876d78c647964f3422f2a" }, "/Users/jfairley/dev/code/storybook-with-cypress/cypress.config.js": { "path": "/Users/jfairley/dev/code/storybook-with-cypress/cypress.config.js", "statementMap": {}, "fnMap": {}, "branchMap": {}, "s": {}, "f": {}, "b": {} }, "/Users/jfairley/dev/code/storybook-with-cypress/stories/Button.stories.js": { "path": "/Users/jfairley/dev/code/storybook-with-cypress/stories/Button.stories.js", "statementMap": {}, "fnMap": {}, "branchMap": {}, "s": {}, "f": {}, "b": {} }, "/Users/jfairley/dev/code/storybook-with-cypress/coverage/lcov-report/block-navigation.js": { "path": "/Users/jfairley/dev/code/storybook-with-cypress/coverage/lcov-report/block-navigation.js", "statementMap": {}, "fnMap": {}, "branchMap": {}, "s": {}, "f": {}, "b": {} }, "/Users/jfairley/dev/code/storybook-with-cypress/coverage/lcov-report/prettify.js": { "path": "/Users/jfairley/dev/code/storybook-with-cypress/coverage/lcov-report/prettify.js", "statementMap": {}, "fnMap": {}, "branchMap": {}, "s": {}, "f": {}, "b": {} }, "/Users/jfairley/dev/code/storybook-with-cypress/coverage/lcov-report/sorter.js": { "path": "/Users/jfairley/dev/code/storybook-with-cypress/coverage/lcov-report/sorter.js", "statementMap": {}, "fnMap": {}, "branchMap": {}, "s": {}, "f": {}, "b": {} }, "/Users/jfairley/dev/code/storybook-with-cypress/cypress/e2e/button.cy.js": { "path": "/Users/jfairley/dev/code/storybook-with-cypress/cypress/e2e/button.cy.js", "statementMap": {}, "fnMap": {}, "branchMap": {}, "s": {}, "f": {}, "b": {} }, "/Users/jfairley/dev/code/storybook-with-cypress/stories/Button.jsx": { "path": "/Users/jfairley/dev/code/storybook-with-cypress/stories/Button.jsx", "statementMap": {}, "fnMap": {}, "branchMap": {}, "s": {}, "f": {}, "b": {} }, "/Users/jfairley/dev/code/storybook-with-cypress/stories/button.css": { "path": "/Users/jfairley/dev/code/storybook-with-cypress/stories/button.css", "statementMap": {}, "fnMap": {}, "branchMap": {}, "s": {}, "f": {}, "b": {} } }
-
- Do you have any custom NYC settings in
package.json(nycobject) or in other NYC config files-
open `.nycrc`
{ "all": true, "include": ["stories/**/*"] }
-
- Do you run Cypress tests in a Docker container?
- no
Describe the bug
I run Cypress e2e tests against my instrumented Storybook stories for React.
After upgrading to Storybook 7, when I run Cypress, I see window.__coverage__ as expected, but .nyc_output/out.json never shows coverage within it.
This worked with Storybook 6, but only upgrading to Storybook 7 breaks cypress coverage.
I assume the problem might be with conflicting transitive dependencies, but I don't know.
Link to the repo
I have created a new repository to showcase the issue. My application repositories are private.
Through local debugging, I've verified that win.__coverage__ is not available when the coverage plugin is running.
It always returns here:
Lines 70 to 73 in 1468c58
Yet... I've added a test step to show that the coverage object can be obtained within the test function.
cy.window().then((win) => {
cy.log("coverage object", win.__coverage__);
});(source)
The best I can tell, __coverage__ just isn't available yet when called from beforeEach, but I have found that it is available during an afterEach.
My tests are working correctly with coverage when I change the beforeEach to an afterEach.
Line 63 in 1468c58
- beforeEach(() => {
+ afterEach(() => {
I don't know what the impacts are for this change. Maybe there can be both a beforeEach and an afterEach. It looks like you already have code that dedupes coverage numbers.
FWIW, I also have this problem when I render Storybook with the Vite builder. (I'm using the default Webpack 5 builder.)
For the short term, I've added the following monkey patch to my repos to get me rolling again. 😭
package.json : "scripts"
"postinstall": "perl -i -pe's/beforeEach/afterEach/g' node_modules/@cypress/code-coverage/support.js",
I replicated this issue and will forward it to the appropriate team. They will evaluate the priority of this ticket and consider their capacity to pick it up. Please note that this does not guarantee that this issue will be resolved. The ticket will indicate status changes during evaluation, so we ask that you please refrain from asking for updates. Thanks!
Are you producing source maps?
Are you producing source maps?
Yes. Coverage works with afterEach. This is a timing issue where __coverage__ isn't yet defined when the storybook beforeEach runs.
For SPA app in general, window.__coverage__ will most likely not exist at the time of window's load event, because instrumented js is loaded asynchronously.
Changing the timing to afterEach instead should be the way to go, unless there is a good reason why it was done with window's load event in the first place.