`skipWhen()` doesn't work as expected if condition has been true once before
bttger opened this issue · comments
The skipWhen condition works as long as the first test hasErrors. After it has no errors anymore the second test runs. This is as expected so far.
But what I did not expect is that the second test continues to run with further inputs when the first test hasErrors again. I would expect that it gets skipped because the skipWhen condition is true again.
import { create, test, enforce, skipWhen } from 'vest';
export const signUpSuite = create((data = {}) => {
test('someField', 'lt3', () => {
enforce(data.someField).longerThan(3);
});
skipWhen(
(res) => res.hasErrors('someField'),
() => {
test('someField', 'ends with x', () => {
enforce(data.someField).endsWith('x');
});
}
);
});Environment:
Node: 16.11
Vest: 4.0.0-next-6abc96
Hey @bttger, thanks for reaching out.
I just tried to run a similar experiment and was unable to reproduce, but maybe I misunderstood the scenario.
Here's my sandbox: https://codesandbox.io/s/react-issues-737-xen0u?file=/src/suite.js
Not sure what framework you're using so I just used the react template. Can you fork the sandbox so it introduces the same result?
In the sandbox I added an alert any time the second test called. I didn't see the alerts firing the second time after the first test was invalid again.
The same thing happens in the sandbox that you provided. You can see it when joining the messages array in the components/Input/index.js file (e.g. {messages.join(" // ")} in line 19).
https://codesandbox.io/s/react-issues-737-forked-ocp8l?file=/src/components/Input/index.js
You can enter e.g. 'ddd' (error: too short), 'ddddd' (no error), 'dddddcat' (error: cat not allowed), 'dcat' (error: both errors, tho the cat error should be skipped)
Ok. Now I understand.
Yes, there seem to be some unexpected behavior here, but one thing I can tell you for certain - the field is indeed being skipped.
What you're seeing is a bit of an odd case, and I guess I'll need to figure out how to solve it. It is somewhat tricky. skipWhen behaves like the broader skip, only focused - meaning:
The field is indeed being skipped during your secondary run, which then prevents it from being re-evaluated, so it retains that previous result that was present inside. This is usually a good thing and very much expected because you want to retain the errors of the skipped fields, and not have them reset when you go out and validate a different field.
Your case is the exception where caching the result can come back to bite us.
Your scenario makes me wonder whether skipWhen should really behave like skip in the sense that it retains the previous validation result, or maybe it should behave more like an omit in which it just drops the whatever's inside as long as the condition is truthy.
Making this change might also play part in solving #735
If I release a testing version of vest with such a change applied, would you be willing to test it and report back? I might be able to have one published tomorrow.
Ah ok, then it makes sense why the alert is indeed being skipped but the errors still stay the same. And yes, I then agree that it would probably make more sense to handle it like an omit case rather than skipping evaluation and caching the errors. Yes, just let me know and I'll test it.
@bttger
I just released vest@4.0.0-dev-fc2efe. I also updated the sandbox to use this version: https://codesandbox.io/s/react-issues-737-xen0u?file=/src/suite.js
You can npm i vest@4.0.0-dev-fc2efe to install it.
It seems like in our use-case, it works as expected, and matches your desired behavior. I still need to see how it behaves in other situations. This change broke a couple of Vest's unit tests, so I'm going to see if it is really misbehaving, or if it is just the state without the extra error that's breaking the test.
If you're interested in what changed, here's the diff: https://github.com/ealush/vest/pull/738/files#diff-89795442aac4bd7e60d030d68f954a7417250f7130bdb81c08590fc29c1575d0R19-R21
Either way, let me know if that works for you. If it does, it might end up in Vest soon (or we'll find another way to introduce test omission that will satisfy this use case).
Thank you for your feedback and helping make Vest better 🙏🏻
Thank you for implementing it so fast. I have installed the new version and my forms are working as expected now. In my opinion, this is now the behavior I'd expect from skipWhen and I'd love to see it in Vest 4. Thanks for your work!
I've run the tests locally to see which ones are affected but I can't really see how they are correlated 😅
@bttger thanks for trying this and reporting back. Your feedback is very helpful in determining the future of this API.
I did find the odd case in which this change doesn't play nicely. Here's a sandbox: https://codesandbox.io/s/react-skipwhen-omits-knm31?file=/src/suite.js
Imagine you have a legitimate case for skipping a field during runtime, yet you don't want to completely omit the test, as it would mark it as optional. For example in the sandbox, I wrapped the confirmation test with a skipWhen so it doesn't get validated when the password is invalid to begin with. For as long as the password is invalid, the confirmation is optional, and the form can be submitted.
This can lead to the opposite side of the confusion that you described. When people wish skipWhen to be a focused skip.
I am starting to feel that in terms of design, an applicable solution would be to keep skipWhen as it is (focused skip), but add one of the following:
omitWhen() wrapper
A comparable API to skipWhen. It does exactly what you need, but will retain the ability to satisfy the need for focused-skip.
omitWhen(res => res.hasErrors('username'), () => {
test(...)
});omit() modifier
A complementary API to skipWhen and group. Calling it within any block in Vest, will omit all the tests within that block. It adheres to the conditional specified in skipWhen, so if it is falsy, it won't have any effect.
skipWhen(res => res.hasErrors('username'), () => {
omit(); // this will mark all the tests within this block as omitted and optional.
test(...)
});Both options are equally powerful, and each has its pros and cons. Either way they will reduce most of the ambiguity, because it makes sense that skipWhen is a "sibling" of skip, and that omit (or omitWhen) completely omits the test from the suite.
If you had to use one, which feels more ergonomic to you?
As a side note:
The omit() modifier option makes me think of this issue: #733 where the request was for a similar execution modifier, for eager returns from blocks. I am wondering if there can be other requested modifiers, or these provide the basic building blocks for a suite.
Imagine you have a legitimate case for skipping a field during runtime, yet you don't want to completely omit the test, as it would mark it as optional. For example in the sandbox, I wrapped the confirmation test with a skipWhen so it doesn't get validated when the password is invalid to begin with. For as long as the password is invalid, the confirmation is optional, and the form can be submitted.
This can lead to the opposite side of the confusion that you described. When people wish skipWhen to be a focused skip.
Yes that's right. The odd case is now that the skipWhen condition is put on another field which doesn't get tested because of the only() call. (Though this suite wouldn't work in real life anyways because I can modify the password again after I have inserted the confirmation. They don't equal anymore but the suite is still valid if the confirmation input was correct once.)
Both options are equally powerful, and each has its pros and cons. Either way they will reduce most of the ambiguity, because it makes sense that skipWhen is a "sibling" of skip, and that omit (or omitWhen) completely omits the test from the suite.
I support this. omitWhen feels semantically correct when you have the intention to omit the test case (and the corresponding message) from the suite result if a condition is met during a test run.
Personally I would prefer omitWhen over the omit call inside of another block. I just think the readability is better and since it would have the same signature as the skipWhen function.
Now that I think of it, I guess it could always create some odd cases if you apply a skipWhen or omitWhen condition on a field other than the one that is currently running and if they are dependent (due to the only() call). But maybe I miss something.
@bttger
Thanks for the quick feedback. This really helps me iterate quickly on these ideas.
I released 4.0.0-dev-cc2a13 which introduces omitWhen, and retains skipWhen as it is. I'll add it to vest@next release once I write the tests and update the documentation for it. In the meantime, can you please let me know if it works the way you intend it to?
Here you can see it running on the sandbox: https://codesandbox.io/s/react-issues-737-xen0u?file=/src/suite.js
Now that I think of it, I guess it could always create some odd cases if you apply a skipWhen or omitWhen condition on a field other than the one that is currently running and if they are dependent (due to the only() call). But maybe I miss something.
Yes, I agree, there are always some odd cases on non-focused fields/tests, but most of these cases self correct very easily once the user starts goes back to the field, tries to submit (to validate all).
In case this is mission-critical that these edge cases never happen, a consumer of Vest can take an extra step and make sure these don't happen on their own end by creating more specific skip/omission rules, or even going further and validating multiple fields together, for example:
only(currentField);
if (currentField === "password") only("confirm");and then
skipWhen(res=>res.hasErrors('password'), () => {
test('confirm', ...)
});
This will make both fields validate together, unless the password is invalid. Not perfect, but it demonstrates that things can be done to avoid common issues.
Hm. That idea lead me to think. It might be useful to add a linked fields utility to Vest, that could tell it to validate multiple fields together on some conditions.
import link from 'vest/link';
link.when('password').
add('confirm');But this one can wait.
If you now install vest@next you'll have access to omitWhen, I also updated the docs
Hm. That idea lead me to think. It might be useful to add a linked fields utility to Vest, that could tell it to validate multiple fields together on some conditions.
Good idea, though I wouldn't need it. As you said, I am validating all fields together anyways before submitting. But maybe someone else has the need for it.
Thank you for the quick omitWhen extension. I've updated to the next branch and only had to rename the skipWhen occurrences. It works nice now :)
I just created a PR (ealush/vest-website#1) for explicit mention about the effects on the validation message in the docs.
Thanks for that. I merged your PR over there 🙏🏻
Also, thank you for all the help and helping me quickly iterate over solutions.
Closing this issue for now :)