"spec" strategy does not work in headed mode

[quad5]

Describe the bug
A clear and concise description of what the bug is.

When test file A fails (with failFast enabled), failFast plugin is skipping future test spec that also has failFast enabled.

To Reproduce
Steps to reproduce the behavior

cypress.json
"env": { "FAIL_FAST_PLUGIN": true, "FAIL_FAST_ENABLED": false },

failFast1.js
describe('in failFast1', { "failFast": {"enabled": true} }, function() { it('failFast1', function() { cy.log('testing in failFast1 file'); expect(1).equal(0); }) })

failFast2.js
describe('in failFast2', { "failFast": {"enabled": true} }, function() { it('failFast2', function() { cy.log('testing in failFast2 file'); expect(2).equal(1); }) })

failFast3.js
describe('in failFast3', { "failFast": {"enabled": true} }, function() { it('failFast3', function() { cy.log('testing in failFast3 file'); expect(3).equal(2); }) })

Expected behavior
A clear and concise description of what you expected to happen.

I assume it would only failFast the describe in existing test spec, and not future ones.

Logs
If applicable, add logs to help explain your problem.

** Operating system, Node.js an npm versions, or browser version (please complete the following information):**

• OS: [e.g. Ubuntu 18.04] - Mac
• Node.js: [e.g. 8.11.1] - v12.18.4
• npm: [e.g. 5.6.0] - 6.14.10
• Browser: [e.g. Chrome 73.0.3683] - Electron 87
• failFast: 2.3.2

Additional context
Add any other context about the problem here.

[Javier Brea]

Hi @quad5,

This is the expected behaviour. The failFast.enabled option produces enabling the skip flag, which produces skipping all of the rest of tests. It can be understood as "Skip the rest of tests if any test inside this block fails", not as "Skip the rest of tests inside this block if any test inside this block fails". It does not provide a "scope" for the fail-fast feature, it provides a mechanism to decide what tests should enable the skip flag or not.

If you want to skip only the rest of tests in the current spec file, you can use the FAIL_FAST_STRATEGY environment variable. Set it to spec, and fails in one file will not affect to others.

[quad5]

Hi, not sure if I did it right.

[Javier Brea]

Hi again @quad5 ,

In my previous comments I was supposing that you were talking about headless mode, not about headed mode running all specs together, sorry for that.

In this case, the spec strategy probably is not working because in headed mode the before hook used to reset the skip flag is executed by Cypress only once. So, in the practice, it is like running a single spec. I didn't notice it because the plugin is intended to be used mainly in headless mode (in CI pipelines), and, in fact, the plugin tests are executed only in headless mode. But obviously this is an unexpected behaviour and it can be considered as a bug. For the moment, I will add a caveat to the documentation and I will try to find a way to make it work also in headed mode.

Thanks for your feedback! 🙂

[Javier Brea]

If the after:spec event is executed more than once in headed mode, then it could be used to reset the skip flag. It needs further investigation.

[Mithun]

Hi, is there a solution for this? We are also facing the same problem. The first test case failing is skipping all the other tests and specs as well which is not expected. Only the remaining tests in the same spec should be skipped and run other specs.

Any workaround for this?

[Javier Brea]

Hi @hansiemithun , I suppose that you are talking also about running Cypress in headed mode. Does it work if you run your tests in headless mode? Have you configured properly the "strategy" as "spec"? Which version of Cypress are you using?

[Mithun]

@javierbrea - when i tried to configure as spec in environmental variable, the other tests in the same suite executes.

I have this inline configuration, can we set up spec here internally? This was noticed in headless mode itself.

 it(
    'Verify pre-requisites',
    {
      failFast: {
        enabled: true
      }
    },
    () => {
      common.goToDashboard()
    }
  );

Cypress version:

"cypress": "9.4.1",
  "cypress-fail-fast": "5.0.0",
  "cypress-wait-until": "1.7.2",

Node version

v14.18.1

Environment

Mac OS Monetary v12.4

[Javier Brea]

@hansiemithun , That configuration only has sense if you have the plugin completely disabled using environment variables and you want to enable it only for that test: https://github.com/javierbrea/cypress-fail-fast#configuration-by-test.

If you want to skip all other tests in the same spec file whenever one fails, then you should have this environment variables in your cypress.json file:

{
  "env":
  {
    "FAIL_FAST_STRATEGY": "spec",
    "FAIL_FAST_ENABLED": true
  }
}

[Mithun]

Thanks, @javierbrea - it's working. I thought we cannot configure it in multiple places and hence felt it was not working...

[Mithun]

@javierbrea - what exactly is FAIL_FAST_BAIL? Is this retry option? We also need to retry for 'n' of times before we skip the other tests in the suite

[Javier Brea]

@javierbrea - what exactly is FAIL_FAST_BAIL? Is this retry option? We also need to retry for 'n' of times before we skip the other tests in the suite

Please open a different issue for that question. It is not related with this one.