Skip remaining field tests as soon as one of them fails

Question

Skip remaining field tests as soon as one of them fails

gbcreation opened this issue 3 years ago · comments

Currently, when we want to skip following tests for a field when one of its tests fails, we have to use skipWhen() multiple times like this:

import { create, test, enforce, skipWhen } from 'vest';
const suite = create('user_form', (data = {}) => {

  test('username', 'Username is required', () => {
    enforce(data.password).isNotEmpty();
  });

  skipWhen(suite.get().hasErrors('username'), () => {
    test('username', 'Username is weak', () => {
      enforce(data.password).longerThan(8);
    });
  });

  skipWhen(suite.get().hasErrors('username'), () => {
    test('username', 'Username is too long', () => {
      enforce(data.password).shorterThan(50);
    });
  });

  test('password', 'Password is required', () => {
    enforce(data.password).isNotEmpty();
  });

  skipWhen(suite.get().hasErrors('password'), () => {
    test('password', 'Password is weak', () => {
      enforce(data.password).longerThan(8);
    });
  });

  skipWhen(suite.get().hasErrors('password'), () => {
    test('password', 'Password is too long', () => {
      enforce(data.password).shorterThan(50);
    });
  });

});
export default suite;

It would be useful to have a less boilerplate way to do that.

Two solutions suggested here:

1°) A general purpose solution could be to have a function named like breakOn(condition, testCallback), where the condition would be evaluated after every test() execution in the testCallback.

For example :

import { create, test, enforce, skipWhen } from 'vest';
const suite = create('user_form', (data = {}) => {

  breakOn(suite.get().hasErrors('username'), () => {

    test('username', 'Username is required', () => {
      enforce(data.password).isNotEmpty();
    });

    test('username', 'Username is weak', () => {
      enforce(data.password).longerThan(8);
    });

  });

  breakOn(suite.get().hasErrors('password'), () => {

    test('password', 'Password is required', () => {
      enforce(data.password).isNotEmpty();
    });
  
    test('password', 'Password is weak', () => {
      enforce(data.password).longerThan(8);
    });

  });

});
export default suite;

Or to stop at the first error:

import { create, test, enforce, skipWhen } from 'vest';
const suite = create('user_form', (data = {}) => {

  breakOn(!suite.get().isValid(), () => {

    test('username', 'Username is required', () => {
      enforce(data.password).isNotEmpty();
    });

    test('username', 'Username is weak', () => {
      enforce(data.password).longerThan(8);
    });

    test('password', 'Password is required', () => {
      enforce(data.password).isNotEmpty();
    });
  
    test('password', 'Password is weak', () => {
      enforce(data.password).longerThan(8);
    });

  });

});
export default suite;

2°) Another alternative solution, based on features provided by most test frameworks, would be to have a beforeEach(callback) hook that could be used globally or in a test group. The callback would be called before each test, and if the returned value is falsy, the test is skipped.

For example:

import { create, test, enforce, skipWhen } from 'vest';
const suite = create('user_form', (data = {}) => {

  group('username tests', () => {

    // Executed before each test of this group
    beforeEach(() => !suite.get().hasErrors('username'));

    test('username', 'Username is required', () => {
      enforce(data.password).isNotEmpty();
    });

    test('username', 'Username is weak', () => {
      enforce(data.password).longerThan(8);
    });

  });

  group('password tests', () => {

    // Executed before each test of this group
    beforeEach(suite.get().hasErrors('password'), () => {

    test('password', 'Password is required', () => {
      enforce(data.password).isNotEmpty();
    });
  
    test('password', 'Password is weak', () => {
      enforce(data.password).longerThan(8);
    });

  });

});
export default suite;

Other example:

import { create, test, enforce, skipWhen } from 'vest';
const suite = create('user_form', (data = {}) => {

  // Executed globally before each test
  beforeEach(() => !suite.get().isValid())

  test('username', 'Username is required', () => {
    enforce(data.password).isNotEmpty();
  });

  test('username', 'Username is weak', () => {
    enforce(data.password).longerThan(8);
  });

  group('password tests', () => {

    test('password', 'Password is required', () => {
      enforce(data.password).isNotEmpty();
    });
  
    test('password', 'Password is weak', () => {
      enforce(data.password).longerThan(8);
    });

  });

});
export default suite;

Evyatar · Answer 1 · Wed Dec 08 2021 23:40:54 GMT+0800 (China Standard Time)

Hi @gbcreation, thanks for the feedback and suggestions.
This correlates with a plan I have for adding an "eager" support, which is intended to behave similarly to the way you require:

const suite = create((data = {}) => {
	eager();

	test('username', 'username is required', () => {
		enforce(data.username).isNotBlank();
	});

	test('username', 'username is too short', () => {
		enforce(data.username).longerThan(2);
	});
});

Calling eager in this scenario would make it so that tests of the same name will be skipped if any of them already failed.

However, each of your approaches have their benefits, which I like.

breakOn

breakOn allows a general purpose condition stopping execution. Could be due to result of a test, or due to some other condition. This is pretty powerful, and can be used in variety of situations.

breakOn's drawbacks

The conditional is a boolean value, it cannot be re-evaluated at every run. Luckily this is easy enough to fix by using a function as the conditional:

breakOn(() => suite.get().hasErrors('password'), () => {
	test(/*...*/)
})

breakOn introduces another level of nesting for tests, when factoring in group, skipWhen and each it can get pretty deep. I'd like to avoid as much as I can adding APIs the nest tests unless it is really worth it (which I don't say it doesn't!).

beforeEach

This still needs some tweaking, I don's see there's a clear relationship between the return value of this function and the instruction to vest to break the current execution - this might make it harder for people to understand how it works. I do think, though, it can lead the way to a more general purpose beforeEach for performing pre-test actions.
It also adds a really nice touch because it can be added at any nesting level, without nesting more tests under it.

As it seems, each approach (including mine) has its own benefits and drawbacks. I'd like to take a couple of days to draft a few ideas, and share them here in this thread, because I obviously see the value such a solution would bring. In the meantime, I'll try to come up with a workaround you could use that would bridge that gap.

Gontran Baerts · Answer 2 · Wed Dec 08 2021 23:47:01 GMT+0800 (China Standard Time)

Yes, you're perfectly right! breakOn() must take a function as first parameter. Thanks for correcting me.

Gontran Baerts · Answer 3 · Wed Dec 08 2021 23:50:03 GMT+0800 (China Standard Time)

Regarding beforeEach(), the name is indeed confusing. Maybe something like shouldRunTest()?

Evyatar · Answer 4 · Thu Dec 09 2021 05:40:48 GMT+0800 (China Standard Time)

Yes, this is better suited, for this scenario. I am thinking if this discussion will lead to some sort of a test interception api, excluding tests before and after their run, and maybe modifying their outputs.

While I'm still looking into it, I am wondering if this consumer-land approach might work for you:

import {create, test, enforce, skipWhen} from 'vest';

const suite = create((data = {}) => {
  const test = conditionalTest(res => res.hasErrors('username'))
  
  test('username', 'username is required', () => {
    enforce(data.username).isNotEmpty()
  })
  
  test('username', 'username is too short', () => {
    enforce(data.username).longerThan(2)
  })
})

function conditionalTest(conditional) {
  return () => {
    skipWhen(conditional(suite.get()), () => {
      test(...args)
    })
  }
}

If you're after more granularity, you can do this within every group, for example.

I tested it, and it seems to be able to do everything you need. It might give us some time to discuss the optimal solution and still not block you.

Evyatar · Answer 5 · Sun Dec 12 2021 07:55:31 GMT+0800 (China Standard Time)

@gbcreation Here's a thought.

What if skipWhen could work both upwards and downwards:

When provided a callback, skipWhen skips whatever's in the callback. When a callback is not provided, it behaves as your suggested shouldRunTest - skips all the tests in the current scope.

I like it because it doesn't require adding new interfaces to Vest, but I dislike it because it makes the same API do two (similar, but) different things. I'll fiddle with it a bit more.

Evyatar · Answer 6 · Sun Feb 20 2022 04:17:02 GMT+0800 (China Standard Time)

@gbcreation, not sure if you're still interested in this feature, but I will be soon adding the "eager" mode functionality #793, I believe your breakOn suggestion works in the "plugin" realm. I am researching the possibility of allowing consumers to "intervene" at certain points in the suite and maybe make alternative decisions.