Outdated README warning for parallel re-runs
MikeMcC399 opened this issue · comments
The Warning paragraph in the README > Parallel section does not reflect how Cypress Cloud currently handles re-runs. It also does not distinguish between the different modes of re-running a workflow in the GitHub Actions UI ("Run workflow" used again, "Re-run jobs" > "Re-run failed jobs" and "Re-run jobs" > "Re-run all jobs").
The Warning text is currently:
Warning
⚠️ : Cypress actions useGITHUB_TOKENto get the correct branch and the number of jobs run, making it possible to re-run without the need of pushing an empty commit. If you don't want to use theGITHUB_TOKENyou can still run your tests without problem with the only note that Cypress Cloud's API connects parallel jobs into a single logical run using GitHub commit SHA plus workflow name. If you attempt to re-run GitHub checks, Cypress Cloud thinks the run has already ended. In order to truly rerun parallel jobs, push an empty commit withgit commit --allow-empty -m "re-run checks" && git push. As another work around you can generate and cache a custom build id, read Adding a unique build number to GitHub Actions.
- If "Run workflow" is used to repeat the run of a workflow with no changes in it, and
GITHUB_TOKENis passed, then the workflow run is correctly recorded in Cypress Cloud. - If "Re-run jobs" > "Re-run failed jobs" is used then the workflow run will be repeated and recorded in Cypress Cloud, however the full set of Cypress specs are then run in the number of containers which contained failed Cypress specs. "Re-run failed jobs" does not lead to re-running only failed Cypress specs. This needs to be explained.
This text needs to be reworked.
It would also be good to add references to the following capabilities of Cypress Cloud in the context of re-running failed Cypress specs:
Related issues
I'm not sure if I have enough information to rework the warning, since it describes in part how Cypress Cloud works and this is not open source, so I would be limited to reverse engineering, which is not the ideal basis to provide definitive documentation.
The first step would be in any case for the Cypress (Cloud) team to review this part of the documentation.
@ryanpei Are you still involved in this area, and if so, could you comment?
Thanks @MikeMcC399 , I will try to find someone on our side who knows the exact behavior here. I'm not sure if "Re-run jobs" (as opposed to just the failed jobs, specifically) would re-run only on the containers which had failed. But otherwise this sounds right.
Yes, @MikeMcC399 's suggested edits are correct
Thanks for your confirmation! Unfortunately since I haven't really understood the original warning text I have great difficulty in revising it sensibly, so I'm not able to resolve this issue on my own.
@MikeMcC399 We used to have issues where users would click 'Rerun jobs' in GitHub Actions and the run would look like it's recording and running, but it essentially just exits 0 and looks like the tests passed, but they didn't run at all. That was because a new unique ci-build-id wasn't generated for each rerun.
If you're not seeing this behavior anymore (that all tests rerun in all situations of rerunning in github-action) then this may have been addressed on the Cloud side. I know they did do some work around better detecting unique identifiers of reruns.
If you're seeing the correct behavior on reruns, we can just remove the entire warning section.
-
Thanks for your feedback! I have not seen any evidence of the problem you describe still being in existence so I have gone ahead and submitted a PR #1113 which removes the warning section and replaces it with advice about avoiding "Re-run jobs" > "Re-run failed jobs". It also adds a reference to Cloud options:
Note that
is still open.