Allow to disable fail-fast in all tests except those specifically enabled

[quad5]

Describe the bug
A clear and concise description of what the bug is.

I have followed steps to install cypress-fail-fast in the Readme section, and yet tests were continue to get executed.

To Reproduce
Steps to reproduce the behavior

describe.only("suite", { "failFast": {"enabled": true} }, function() {
it('test1', function() { // do something})
it('test2', function() {// do something})
})

Expected behavior
A clear and concise description of what you expected to happen.

I have only configured failFast option in 1 describe block as seen in above codes. I would have expected test2 and the rest in that describe block to not get executed when test1 failed.

Logs
If applicable, add logs to help explain your problem.

** Operating system, Node.js an npm versions, or browser version (please complete the following information):**

• OS: [e.g. Ubuntu 18.04] - Mac 11.0.1
• Node.js: [e.g. 8.11.1] - node: '12.18.4'
• npm: [e.g. 5.6.0] - npm: '6.14.10',
• Browser: [e.g. Chrome 73.0.3683] - Chrome: 87.0.4280.141
• Cypress: tried 5.6.0 & 6.2.1

Additional context
Add any other context about the problem here.

[Javier Brea]

Hi @quad5, have you also enabled the plugin using the CYPRESS_FAIL_FAST environment variable as described in the "usage" section of the docs?

[quad5]

Maybe I am misunderstanding something.

When I wrote this ticket, I had "env": { "FAIL_FAST": false} in cypress.json and { "failFast": {"enabled": true} } in a describe. What I assume it would do is to fail fast only on the describe which I enabled failFast, and the rest of the other spec files do not fail fast. This is how I want it to behave.

But that doesn't seem to be the case. What is your suggestions on getting this scenario to work? I want to avoid enabling/disabling each individual test/suite.

[Javier Brea]

Well, for the moment the only thing you can do is to specifically disable the rest of tests using the { "failFast": {"enabled": false} } option, and enable the one you want with { "failFast": {"enabled": true} }.

The intention of the CYPRESS_FAIL_FAST environment variable is to allow to totally disable the plugin, ignoring even the options declared programatically. This can be useful for local environments, for example. It can be a little confusing, you are right, so a possible solution could be to rename this option to CYPRESS_FAIL_FAST_PLUGIN, set it to true by default (as described in #44), and add a new variable named CYPRESS_FAIL_FAST_ENABLED, which would work as you expected, enabling/disabling failFast in all tests, and allowing to override this value only in the desired tests.

Thanks for your feedback, and please let me know your opinion about this proposal 🙂

[quad5]

I am assuming that CYPRESS_FAIL_FAST and "env": { "FAIL_FAST": false} in cypress.json does the same thing.

I am not sure what I am about to propose is an efficient or elegant way but here goes.

I would have CYPRESS_FAIL_FAST disabled by default and rename it to CYPRESS_FAIL_FAST_GLOBAL. Rename failFast to failFastLocal

Your note on Readme: (Note that setting this property as true will not have effect if the plugin is disabled globally using the FAIL_FAST environment variable). I would change it to have individual failFast enabling feature for only such test/suite when CYPRESS_FAIL_FAST_GLOBAL is set to false.

Here are some scenarios with my above proposal:

• CYPRESS_FAIL_FAST_GLOBAL is disabled by system (meaning variable isn't declared or set to false in cypress.json ).
• no individual failFastLocal declaration
• Behavior: fail fast feature has no effect on framework meaning test continues to run even with previous failure.

• CYPRESS_FAIL_FAST_GLOBAL is disabled by system (meaning variable isn't declared or set to false in cypress.json).
• individual failFastLocal declaration in test/suite
• Behavior: fail fast feature applies only to individual failFastLocal declaration in test/suite. In other specs/test/suite, feature doesn't apply.

• CYPRESS_FAIL_FAST_GLOBAL is enabled by system (meaning its setting is being set to true in cypress.json).
• no individual failFastLocal declaration
• Behavior: fail fast feature is in effect in all specs

• CYPRESS_FAIL_FAST_GLOBAL is enabled by system (meaning its setting is being set to true in cypress.json).
• individual failFastLocal declaration
• Behavior: fail fast feature applied to all specs, and to individual failFastLocal as how they are declared.

[Javier Brea]

Thanks for the proposal. I think it is right, and very similar to the mine's one (maybe I didn't explain it well enough), but with little differences:

• In my opinion there is no need to rename failFast as failFastLocal. If the "global" variable is named FAIL_FAST_ENABLED, as I suggested, I think it would be clear enough to understand that FAIL_FAST_ENABLED=true is a global value for {failFast: {enabled:true}}
• The current FAIL_FAST environment variable will be still needed, as it can be necessary to disable the plugin totally for some scenarios, as local development, etc. I suggested to rename it to FAIL_FAST_PLUGIN for major clarity on what it does.
• I think it is better to give true default values to both variables, so the default behavior of the plugin will be to stop on first fail once installed, without the need of defining any specific setting.

So, following your scenarios examples, here is how it would work:

1. Default values of plugin when installed

   • CYPRESS_FAIL_FAST_PLUGIN=true (default, no need to define it)
   • CYPRESS_FAIL_FAST_ENABLED=true (default, no need to define it)
   • No individual failFast declaration
   • Behavior: Fail fast feature is applied to all tests.

2. User wants to apply fail fast feature only to specific tests

   • CYPRESS_FAIL_FAST_PLUGIN=true (default, no need to define it)
   • CYPRESS_FAIL_FAST_ENABLED=false (defined by user)
   • Individual {failFast: {enabled:true}} configuration in the test/describe the user wants to apply fail fast.
   • Behavior: Fail fast feature is not applied to any test, except those specifically enabled.

3. User wants to disable fail fast feature only on specific tests

   • CYPRESS_FAIL_FAST_PLUGIN=true (default, no need to define it)
   • CYPRESS_FAIL_FAST_ENABLED=true (default, no need to define it)
   • Individual {failFast: {enabled:false}} configuration in those tests in which fail fast feature has not to be applied
   • Behavior: Fail fast feature is applied to all tests, except those specifically disabled.

4. User wants to disable fail fast feature in all tests, even in those specifically enabled using individual configuration

   • CYPRESS_FAIL_FAST_PLUGIN=false (defined by user)
   • CYPRESS_FAIL_FAST_ENABLED=true (default, no need to define it)
   • Individual {failFast: {enabled:true}} configuration in those tests in which fail fast feature has to be applied
   • Behavior: Fail fast feature is never applied

[quad5]

As for me, I was trying to avoid creating extra variable. Maybe it is because I don't understand what FAIL_FAST_PLUGIN variable does honestly. It is probably because I don't know CI well enough to understand the important of having that variable.

[Javier Brea]

As for me, I was trying to avoid creating extra variable. Maybe it is because I don't understand what FAIL_FAST_PLUGIN variable does honestly. It is probably because I don't know CI well enough to understand the important of having that variable.

Yes, it is mainly related to allow to enable/disable totally fail-fast on CI and/or local environments without having to change code. As it can be enabled/disabled programmatically for some tests, a mechanism to ignore {failFast: {enabled:true}} declarations is needed, just in case you want to never fail fast in some environments. That's what FAIL_FAST_PLUGIN does. Anyway, maybe it would be used only in complex scenarios, so it can be considered as an advanced configuration, and, as its default value is true, it won't disturb to users working on simpler systems.

[quad5]

Would you be able to see if CYPRESS_FAIL_FAST_PLUGIN exists or not? if it doesn't exist, then use fail fast feature by its default value. if it exists, then use the feature based on its specification? I am trying to see if we can combine CYPRESS_FAIL_FAST_PLUGIN and CYPRESS_FAIL_FAST_ENABLED. Just a suggestion.

[Javier Brea]

I can't see how to combine them, as they work in a different way. But, if CYPRESS_FAIL_FAST_PLUGIN is not defined, then its default value will be true, so it will have no effect. In the practice, it will only be useful when explicitly defined and set to false, because all of the rest of settings and options will be ignored and fail-fast will never be applied. I don't know if I'm explaining it well... maybe we are trying to explain a very similar behavior but with different words.

[quad5]

Sorry I don't mean to be annoying. My last clarification, I promise :)

here is what i understand from your's proposal:
FAIL_FAST_PLUGIN - turn plugin on/off. if it is off, feature won't be applied even if failFast: { enabled: true }
FAIL_FAST_ENABLED - turn feature on/off. if plugin is off, then feature should be off. if plugin is on, then feature is based on ENABLED & failFast

here is what I was thinking
We consider FAIL_FAST_ENABLED as FAIL_FAST_PLUGIN. if FAIL_FAST_ENABLED variable doesn't exist in environment, we consider feature as off. if FAIL_FAST_ENABLED exists, then feature is based on its value & failFast.

[Javier Brea]

Ok, sorry, now I understood. Don't worry, you're not annoying 🙂

You also understood my proposal well. The problem I can see in your proposal is that the plugin would be disabled by default, requiring to declare FAIL_FAST_ENABLED as true or false, otherwise the whole plugin will be disabled, and, in my opinion, it would be better to enable it by default once installed, without the need to declare any environment variable, as it is described in #44.

With my proposal, to get your scenario working you will only need to declare FAIL_FAST_ENABLED=false, and apply specific failFast: { enabled: true } configuration in those tests you want to apply fail-fast.

[quad5]

Ok, sorry, now I understood. Don't worry, you're not annoying 🙂

You also understood my proposal well. The problem I can see in your proposal is that the plugin would be disabled by default, requiring to declare FAIL_FAST_ENABLED as true or false, otherwise the whole plugin will be disabled, and, in my opinion, it would be better to enable it by default once installed, without the need to declare any environment variable, as it is described in #44.

With my proposal, to get your scenario working you will only need to declare FAIL_FAST_ENABLED=false, and apply specific failFast: { enabled: true } configuration in those tests you want to apply fail-fast.

👍 . Thanks for implementing this btw.