[Bug] `waitFor` is not working when imported from @storybook/testing-library
IanVS opened this issue · comments
Describe the bug
I have a story which is testing a validation error. It triggers the error, waits for an element with a role of alert, and then tabs out of the field so I can snapshot a red border on the input. This works fine if I import directly from @testing-library, but I get an error of Unable to find role="alert", and the DOM it prints out sure enough does not include the alert. But it occurs almost immediately, even if I try to add a large timeout, such as await waitFor(() => screen.getByRole('alert'), { timeout: 6000 });.
Steps to reproduce the behavior
Trigger an action that takes a moment to add an element, and use waitFor to proceed only after that element is found.
Expected behavior
The behavior should match that of waitFor from @testing-library/react.
Screenshots and/or logs
N/A
Environment
- OS: mac
- Node.js version: 16.4.0
- NPM version: 8.1.3
- Browser (if applicable): Brave
- Browser version (if applicable): [e.g. 22]
- Device (if applicable): [e.g. iPhone6]
Additional context
Let me know if I should create a reproduction for this.
I've tried adding addon-interactions, in case somehow storybook was relying on that being available, but it does not solve this issue.
Could you share the entire play function?
@ghengeveld sure thing, here's a reproduction: https://github.com/IanVS/vite-testing-example/blob/80886928fca6ad50e06dbbda2a8d105f73513848/src/screens/animals/AnimalsPage.stories.tsx#L26
The only difference in this branch is using @storybook/testing-library rather than @testing-library directly. The main branch works great, but this branch fails.
We are experiencing the same issue
This seems to be caused by having multiple instances of @testing-library/dom installed.
If you are using yarn, here is an easy way to check:
yarn why @testing-library/dom
It will be interesting to see the reason why this causes the issue and if it can be avoided even when multiple instances are installed. In my case hoisting did not work for some weird reason and a reinstall helped.
Looks like I only have one of them:
▶ npm why @testing-library/dom
@testing-library/dom@8.10.1
node_modules/@testing-library/dom
@testing-library/dom@"^8.3.0" from @storybook/testing-library@0.0.7
node_modules/@storybook/testing-library
dev @storybook/testing-library@"^0.0.7" from the root project
@testing-library/dom@"^8.1.0" from @testing-library/cypress@8.0.2
node_modules/@testing-library/cypress
optional @testing-library/cypress@"^8.0.2" from the root project
@testing-library/dom@"^8.0.0" from @testing-library/react@12.1.2
node_modules/@testing-library/react
dev @testing-library/react@"^12.1.2" from the root project
peer @testing-library/dom@">=7.21.4" from @testing-library/user-event@13.5.0
node_modules/@testing-library/user-event
dev @testing-library/user-event@"^13.5.0" from the root project
@testing-library/user-event@"^13.2.1" from @storybook/testing-library@0.0.7
node_modules/@storybook/testing-library
dev @storybook/testing-library@"^0.0.7" from the root project
Ignore my previous comment.
Some new observations - actually the problem is not in wait, but in the way the functions are wrapped. The original functions will throw an error if the element was not found, while the wrapped ones will return the error. This will confuse waitFor as it waits until the callback stops throwing.
Ah, I wonder if that helps to explain #8 as well!
I tried naively changing the return to a throw, but then it enters a loop. I think maybe the call inside waitFor should not be intercepted, but I don't see an easy way to test that idea so far.
Any update on this? I find I'm having to avoid writing certain tests as this makes my tests flaky. Sometimes it find the element and other times it errors so I can't rely on it.
I've been using waitFor directly from @testing-library/react, but that means I can't use the interactions addon, which is a bummer. But at least my tests work as expected.
Bump. This is still a problem. It's not reproducible in a non-storybook environment so instrumented versions must be getting activated only in a storybook env.
For example see the following 2 tests. They behave the same outside storybook env but behave differently in storybook env. Interactions fail in the first but they pass for the latter.
Instrumented getByRole returns an error instead of throwing it. The test will throw an error in the next line when trying to call el.hasAttribute. But it seems waitFor doesn't catch it. So the error propagates to the react rendering and a react error overlay is displayed. So I guess instrumented waitFor doesn't work with synchronous errors thrown.
const test = async () => {
const el = await waitFor(() => {
const el = screen.getByRole('button')
// interactions fail with err that el.hasAttribute is not a function
if (el.hasAttribute('disabled')) {
throw new Error('should be enabled')
}
return el
})
await userEvent.click(el)
}
vs.
const test = async () => {
const el = await waitFor(async () => {
// this version works fine because there are no sync errors thrown
const el = await screen.findByRole('button')
if (el.hasAttribute('disabled')) {
throw new Error('should be enabled')
}
return el
})
await userEvent.click(el)
}
If this will not be fixed (maybe because a limitation of instrumentation), then we should reeducate ourselves and our teams to use waitFor according to this library's expectations, so please do let us know.
BTW upgrading to v7 was far from trivial so we gave up on that for now.
@anilanar what issues did you have in the 7.0 migration? We're trying to make it as easy as possible, but there are indeed a fair number of changes.
@IanVS We couldn't get past some build errors, most likely related to native esm support but we didn't analyze too deep. We just looked up if somebody already reported them and nobody did. Could be specific to our setup or could be specific to some libraries we use. e.g. might be a library that uses .mjs or package.json exports wrongly. Also hard to make an educated guess without looking deeper into what changes were done on the webpack builder internally in v7. We also have custom webpack/babel configs and we modify loaders, I guess that's typical for any serious/large web app.
@IanVS Can you clarify whether if implementation of waitFor, screen.getBy* etc. come from the storybook environment so that they are tied to storybook version or if their implementation is isolated in @storybook/testing-library package?
I'm not super familiar, honestly, but I do know that both the storybook addon-interactions and this package use @storybook/instrumenter, and therefore are a bit tied to each other. For example, storybook 7 needs to use 0.0.14-next.1 of this package.
@IanVS I analyzed the source code of @storybook/instrumenter and @storybookjs/testing-library. Sadly, they completely change the semantics of *.getBy* and therefore that of waitFor and it's still the same in v7 / 0.0.14-next.1.
I wish that was documented in the first place when interactions addon first emerged; then we wouldn't have adopted this tool, or we would have adopted it in a different manner. That would have saved us tremendous amount of time from debugging flaky/broken tests.
For now, we decided to stop using @storybookjs/testing-library, and I still think differences between @storybookjs/testing-library and normal testing library must be documented in the home page with bold letters.
@anilanar can you explain what you mean with changed semantics? What is it about those functions that Storybook changes in a way that you can't use them? We can tweak and fix things to be more compatible, but right now we don't have sufficient information.
By the way, you can totally use the non-Storybook version of those packages in your play functions. You just won't be able to use the interactions panel.
@ghengeveld Sure.
testing-library's waitFor depends on errors thrown within the callback to decide whether to retry or not. If the closure returns a promise, then it decides based on whether if the promise rejects or not. It retries until callback doesn't throw/reject or until timeout is reached, whichever happens first.
getBy* throws an error if it cannot find anything and findBy* returns a rejected promise if it cannot find anything so that's how waitFor, getBy* and findBy* fits in together.
Based on those semantics, we have additional internal testing utility functions: e.g. one that waits for a button to become enabled, getBy*/findBy* doesn't have such querying capabilities. waitFor(() => { const btn = getBy*(...); invariant(isEnabled(btn)); })
In comparison, instrumented getBy* doesn't throw an error when it doesn't find anything and it's not clear to me how waitFor works in light of that.
I'm not seeing this as a problem anymore, is anyone else still having issues with current versions?
storybookjs/storybook@501b7e1 might be the fix.