Cypress coverage is reseted on each tests instead of getting the full coverage.

Question

Cypress coverage is reseted on each tests instead of getting the full coverage.

samixchoumi opened this issue a year ago · comments

Logs and screenshots
Here is the "problem" i think i've found :

For each test, it reset the coverage, then the index.html from lcov-report is reseted.

Versions

What is this plugin's version? If this is NOT the latest released version can you try the latest version, please?
"@cypress/code-coverage": "^3.10.4",
If the plugin worked before in version X, but stopped after upgrading to version Y, please try the released versions between X and Y to see where the breaking change was.
What is Cypress version?
"cypress": "^12.12.0",
What is your operating system? Mac
What is the shell? -zsh
What is the Node version? v18.12.1
What is the NPM version? 9.6.4
How do you instrument your application? Cypress does not instrument web application code, so you need to do it yourself. I use the command : npx nyc instrument --compact=false src instrumented
When running tests, if you open the web application in regular browser, and open DevTools, do you see window.__coverage__ object? Can you paste a screenshot? Yes
Is there .nyc_output folder? Is there .nyc_output/out.json file. Is it empty? Can you paste at least part of it so we can see the keys and file paths?

{
  "/Users/username/IdeaProjects/projectname/src/app/i18n.ts": {
    "path": "/Users/username/IdeaProjects/projectname/src/app/i18n.ts",
    "statementMap": {
      "0": {
        "start": {
          "line": 9,
          "column": 0
        },
        "end": {
          "line": 9,
          "column": 29
        }
      },
      "1": {
        "start": {
          "line": 10,
          "column": 0
        },
        "end": {
          "line": 10,
          "column": 29
        }
      },
      "2": {
        "start": {
          "line": 11,
          "column": 0
        },
        "end": {
          "line": 11,
          "column": 29
        }
      },
      "3": {
        "start": {
          "line": 12,
          "column": 0
        },
        "end": {
          "line": 12,
          "column": 29
        }
      },
      "4": {
        "start": {
          "line": 13,
          "column": 0
        },
        "end": {
          "line": 13,
          "column": 29
        }
      },
      "5": {
        "start": {
          "line": 14,
          "column": 0
        },
        "end": {
          "line": 14,
          "column": 29
        }
      },
      "6": {
        "start": {
          "line": 15,
          "column": 0
        },
        "end": {
          "line": 15,
          "column": 29
        }
      }
    },
    "fnMap": {},
    "branchMap": {},
    "s": {
      "0": 16,
      "1": 16,
      "2": 16,
      "3": 16,
      "4": 16,
      "5": 16,
      "6": 16
    },
    "f": {},
    "b": {},
    "inputSourceMap": {
      "version": 3,
      "mappings": "AAAA,SAASA,kBAAkB,QAAQ,iBAAiB;AACpD,OAAOC,QAAQ,MAAM,4BAA4B;AACjD,OAAOC,QAAQ,MAAM,4BAA4B;AACjD,OAAOC,QAAQ,MAAM,4BAA4B;AACjD,OAAOC,QAAQ,MAAM,4BAA4B;AACjD,OAAOC,QAAQ,MAAM,4BAA4B;AACjD,OAAOC,QAAQ,MAAM,4BAA4B;AACjD,OAAOC,QAAQ,MAAM,4BAA4B;AAEjDP,kBAAkB,CAACG,QAAQ,CAAC;AAC5BH,kBAAkB,CAACI,QAAQ,CAAC;AAC5BJ,kBAAkB,CAACE,QAAQ,CAAC;AAC5BF,kBAAkB,CAACK,QAAQ,CAAC;AAC5BL,kBAAkB,CAACM,QAAQ,CAAC;AAC5BN,kBAAkB,CAACC,QAAQ,CAAC;AAC5BD,kBAAkB,CAACO,QAAQ,CAAC",
      "names": [
        "registerLocaleData",
        "localeDe",
        "localeEs",
        "localeFr",
        "localeIt",
        "localePl",
        "localePt",
        "localeZh"
      ],
      "sourceRoot": "",
      "sources": [
        "/Users/username/IdeaProjects/projectname/src/app/i18n.ts"
      ],
      "sourcesContent": [
        "import { registerLocaleData } from '@angular/common';\nimport localeDe from '@angular/common/locales/de';\nimport localeEs from '@angular/common/locales/es';\nimport localeFr from '@angular/common/locales/fr';\nimport localeIt from '@angular/common/locales/it';\nimport localePl from '@angular/common/locales/pl';\nimport localePt from '@angular/common/locales/pt';\nimport localeZh from '@angular/common/locales/zh';\n\nregisterLocaleData(localeFr);\nregisterLocaleData(localeIt);\nregisterLocaleData(localeEs);\nregisterLocaleData(localePl);\nregisterLocaleData(localePt);\nregisterLocaleData(localeDe);\nregisterLocaleData(localeZh);\n"
      ]
    },
    "_coverageSchema": "1a1c01bbd47fc00a2c39e90264f33305004495a9",
    "hash": "ceff16556a77183c55957957cc62fdbf5354210a"
  },

Do you have any custom NYC settings in package.json (nyc object) or in other NYC config files
.nycrc config :

{
  "extends": "@istanbuljs/nyc-config-typescript",
  "all": true,
  "exclude": [
    "./coverage/**",
    "cypress/**",
    "./dist/**",
    "**/*.spec.ts",
    "./src/main.ts",
    "./src/test.ts",
    "**/*.conf.js",
    "./instrumented/**"
  ]
}

Do you run Cypress tests in a Docker container? No

Describe the bug
When I launch test, i saw that it launch a before all step (as showed before on the screenshot) that reset the coverage, but the problem i meet is that it reset all the current coverage of the app instead of just the actual test. So after each test, the index.html is specific to the current test instead of all the app coverage.

I searched on the doc but i don't find any issue like me.

I launch the command : cypress open --env configFile=local,TAGS="@testenv and not @ignore", it seem to be the same when I do cypress run command.

Link to the repo
No link to the repo available.

Example
Everything is explained on the Describe the bug section.

samixchoumi · Answer 1 · Wed Jun 07 2023 20:18:13 GMT+0800 (China Standard Time)

@warrensplayer any news ?

Mike Plummer · Answer 2 · Wed Jun 21 2023 03:41:26 GMT+0800 (China Standard Time)

Hi @samixchoumi , thanks for using Cypress! The code coverage plugin will reset coverage before each spec when running in open mode, but when in run mode all specs should be combined into a consolidated coverage report. I set up a quick reproduction project to demonstrate, feel free to review.

Is it possible you're using a plugin or different tech stack that may alter this behavior? Can you try removing your use of TAGS=* and other env params to see if that may be your issue? If you're still only getting partial coverage when using cypress run ... without those customizations can you share your project (or a simple reproduction case) so I can investigate?

samixchoumi · Answer 3 · Wed Jun 21 2023 17:48:25 GMT+0800 (China Standard Time)

Hello @mike-plummer, I'm taking a look at it and will update you as soon as possible !
I didn't know about that. I think that it could be nice to add an option to allow us to activate or not the fact that it reset in open mode.

Little question :
Do you know if it's normal (when doing cypress run) that the index.html is not updated at the end of each spec as it is done for cypress open command ?
If yes, should we wait at the end of the cypress run execution to get our final coverage (and then our index.html fully updated) ?

Mike Plummer · Answer 4 · Wed Jun 21 2023 20:56:03 GMT+0800 (China Standard Time)

@samixchoumi The report is only generated once at the very end of your run for performance reasons - see the code here. What form the report(s) takes depends on what reporters you have configured, but by default it would include the index.html output

samixchoumi · Answer 5 · Wed Jun 21 2023 21:59:20 GMT+0800 (China Standard Time)

Hum, got an error linked to #667, I downgraded cypress version and relaunch a full run to see if it work, because my last try seem to doesn't get the coverage (maybe a config problem that I resolved too because the project has been update a lot since i created this ticket)

samixchoumi · Answer 6 · Thu Jun 22 2023 16:35:34 GMT+0800 (China Standard Time)

It seem to works fine using cypress run commands, thanks for the help !