[Tests] Ensure that axe calls are made on open state of the component
khiga8 opened this issue · comments
Hi! I noticed that in the component snapshot tests, axe scans are made on the default state of the component.
For example, in ActionMenu, the scan axe call is made like so:
test('axe @aat', async ({page}) => {
await visit(page, {
id: 'components-actionmenu-examples--groups-and-descriptions',
globals: {
colorScheme: theme,
},
})
await expect(page).toHaveNoViolations({
rules: {
'aria-required-children': {
enabled: false,
},
},
})
})Axe is being called on the closed state of the menu - Example snapshot, which does not provide as much value because we're essentially running an axe scan on a button, rather than on the menu markup that is only visible open activating the toggle.
Instead (or in addition to what we currently have), I believe we would want the axe scans to happen on the open state of the menu. I observed this in other "popover" type components like Dialog as well.
We should update these tests to have the axe scan run on the open state.
Stretch: It would be great if we could have some kind of automated check that ensures that we have axe scan coverage on the open state of components.
I would love to pick this up as a way to get my feet wet! 👀
@khiga8 sounds great! Wanted to ask you, for the stretch goal is there anything that you all have found helpful for this in other projects like PVC or in github/github?
That's a great question @joshblack! I am not sure the same setup would translate to Primer React, but happy to share what I've observed about setups in other projects.
I've noticed that e2e tests in Primer React are reserved for auto-generated visual regression tests and axe scans tests. It sounds like this is because (as you've noted before), it is possible to write very robust React unit tests that test component behaviors. Since you can test most interactions (e.g. keyboard behavior) with unit tests, it isn't really worth testing these in e2e tests.
If I recall correctly, it's not possible to test JS behaviors with Rails unit tests. In PVC, in order to test (any kind of JS) interactions, e2e browser tests need to be written. Browser tests in PVC are written (manually, not auto-generated) with the intent of testing interactions because unit tests are limited. The e2e tests have been set up in a way where whenever interactions are made (e.g. a click call), an axe scan is automatically called. This expands coverage of axe to different states of the component automatically. It doesn't seem like this same setup would make sense in Primer React though and would require a significant change.
If we want to work with the existing setup, I wonder if a starting point could be to modify the script/generate-e2e-tests.js script, and have this cross-check the component markup (somehow?), and if it is role="dialog", role="combobox", role="menu" (or any markup that "pops over" or has a hidden initial state), the auto-generated tests would include the following:
Pseudo:
test('axe @aat' on open state, async ({page}) => {
await visit(page, {
id: '${story.id}',
globals: {
colorScheme: theme
}
})
// Remove this after addressing TODO
expect(false).toBeTruthy();
// TODO: We want the axe scan to run on the open state of the component. Please replace this with a call to open the component!
await expect(page).toHaveNoViolations()
})Basically, try to call attention to the fact that there should be test coverage on the open state by noting it in the test and having it auto-fail.
Curious what you think of this approach to start with, or if you have any other ideas!
The following components have an open state and sufficient snapshot/axe coverage on it ✅
The following components have an open state and no snapshot/axe coverage on the open state ❌ , but are facing deprecation anyways. So I will take no action for now:
Opening a PR, just for ActionMenu!