Theme test sometimes fails

[Alister Scott]

The theme test sometimes fails,

https://21850-57936731-gh.circle-artifacts.com/0/home/circleci/wp-e2e-tests/screenshots/FAILED-EN-MOBILE-can-see-theme-details-page-and-open-the-live-demo-1533529831243.png

https://circleci.com/gh/Automattic/wp-e2e-tests/21850#artifacts/containers/0

  1) [WPCOM] Switching Themes: (mobile) Switching Themes @parallel @jetpack @visdiff Can switch free themes Can see theme details page and open the live demo:
04:30:31      TimeoutError: Timed out waiting for element with css selector of 'a.theme__sheet-preview-link,li.is-theme-preview a' to be clickable
04:30:31 Wait timed out after 20025ms
04:30:31       at /home/circleci/wp-e2e-tests/node_modules/selenium-webdriver/lib/promise.js:2201:17
04:30:31       at process._tickCallback (internal/process/next_tick.js:68:7)

I believe this is due to Automattic/wp-calypso#25720

[Rachel McR.]

In Automattic/wp-calypso#25720 it looks like the button not being present is because the theme is active on the site already. Note that in your second screenshot in that issue there's a "Customize site" button (which shows when you view the details for your site's active theme) at the top of the page instead of an "Activate this design" button.

Could this be happening because our theme switch tests are running in parallel? (That is, while one test is opening the theme details another test is activating that theme on the site.)

I'll watch for this failure to pop up again — now that we're capturing videos for failed tests, we could verify if that's happening when this test fails.

[Rachel McR.]

The workaround in #1394 is causing this test to start failing on a later step:

https://22025-57936731-gh.circle-artifacts.com/1/home/circleci/wp-e2e-tests/screenshots/videos/can-see-the-theme-thanks-dialog-and-go-back-to-the-theme-details-page-2018-08-10T15-33-40.mpg

https://circleci.com/gh/Automattic/wp-e2e-tests/22025#tests/containers/1

As you can see in the video:

• When the test gets to the Themes page, Twenty Fourteen is listed as the active theme on the site.
• When the test gets to the theme details page for Twenty Fifteen, that's the active theme on the site.
• Clicking the screenshot in the step Can see theme details page and open the live demo still opens the live demo.
• The next step Can activate the theme from the theme preview page clicks the primary button, but since this is the active theme on the site that button opens the customizer instead of activating the theme. (This step really shouldn't be able to pass in this situation, but the button selector in the test is the same regardless of what action is available here.)
• The step Can see the theme thanks dialog and go back to the theme details page fails because now the customizer is open.

On a retry, the test passes, presumably because there isn't another theme test running and switching the theme at the same time as this test.

I think we should revert #1394 and try a different approach to avoid the collisions between theme tests. What do you think @alisterscott?

[Alister Scott]

Thanks for the sleuthing @rachelmcr - the videos really help

I think we should revert #1394 and try a different approach to avoid the collisions between theme tests. What do you think @alisterscott?

Yes, I think we should revert that change

I'm not sure how we make this repeatable in parallel, maybe we shorten the theme tests to only view themes but never actually activate? Switching themes itself is part of WordPress core so we don't really need to be testing core functionality for our Calypso/WordPress.com sites IMO

What do you think?

[Rachel McR.]

I'm not sure how we make this repeatable in parallel, maybe we shorten the theme tests to only view themes but never actually activate?

I think it's worth having one test that activates, to confirm that the correct dialog appears after activation. Right now in wp-theme-switch-spec.js we have two tests:

1. Switching Themes
2. Activating Themes

We could keep the full activation flow (which proceeds to open the customizer) for Activating Themes. I'd propose making the following change to Switching Themes:

• Change the test name to something like Previewing Themes.
• Remove the activation step.
• Preview a completely different theme (one that we know will never be activated)
• Stop once we confirm the live demo appears as expected with an option to activate the theme from there.

That does also remove the check for being able to go back to the theme detail page after activation, but that seems like a minor step in this flow and overall should resolve the problem.

[Rachel McR.]

I can spin up a PR with those proposed changes so we can take a look at how it works.