Any known issues with Cypress 5.x.x? (failed test reported as passed)
cbrunnkvist opened this issue · comments
I recently updated one of my projects to use Cypress 5.2.0 and started seeing tests with failed assertions showing up as green in TeamCity. I have not yet had an opportunity to make a minimal reproducible example, so I'm just throwing this out here: has anyone else noticed any similar issue?
Expected: 5 tests, 4 passing, 1 failing as per TC Build Log excerpt below:
Running: reports.spec.js (3 of 3)
Reports area 1s
(Results)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Tests: 5 │
│ Passing: 4 │
│ Failing: 1 │
│ Pending: 0 │
│ Skipped: 0 │
│ Screenshots: 3 │
│ Video: true │
│ Duration: 18 seconds │
│ Spec Ran: reports.spec.js │
...
Actual: 5 tests, 5 passing, 0 failing as per TC Tests tab screenshot below:
I noticed that there are some issues going on with Cypress 5.2.0. Not sure yet what 🤔
I have few projects with CI running and using Cypress 5.1.0 works fine, but moving to 5.2.0, there is an issue, even in the project that uses only Mocha. I guess, for now, we need to wait for version 5.2.1 or an update from mocha package to have it figured out.
I will keep following this and I will make a update on this package as soon as possible
I'm getting closer: the error now seems to only happen when I run inside a cypress/included Docker container via docker-compose. When I enter the container and run cypress from an interactive shell - newlines are |n as expected. I'd really like to get to the bottom of this! 😅
The exact same thing seems to happen with the bundled "teamcity" reporter as well.
How to recreate: just assert(false, "these\nare\nnewlines") anywhere. Below, I did it in a beforeEach.
Expected:
##teamcity[testFailed name='"before each" hook for "it should work"' message='these|nare|nnewlines|n|nBecause this error occurred during a before eachhook we are skipping the remaining tests in the current suite:when logged in' details='AssertionError: these|nare|nnewlines|n|nBecause this error occurred during a before eachhook we are skipping the remaining tests in the current suite:when logged in|n at Context.eval (https://www.agoda.com/__cypress/tests?p=cypress/integration/login.spec.js:112:7)' captureStandardOutput='true' flowId='53']
Actual:
##teamcity[testFailed name='"before each" hook for "it should work"' message='these
If you look at the Build Log, TeamCity warns that the message was not formatted correctly, but the test failure fails to get recorded, so builds always become green. 🤢
Phew. 😆
I am happy to confirm that this was completely my own fault - an unforeseen side-effect of piping and filtering output of the cypress process - closing.
@cbrunnkvist glad to hear that you have solved ;)
Btw, the package was updated to support cypress 5 and mocha 8 without issues 👍