Fail in before hook are considere as another test case

Question

Fail in before hook are considere as another test case

jeremie-val opened this issue 2 years ago · comments

Bug :
When executing a tests, if the test succeed, it create a test case on allure side.
If the test fail in the before hook, it consider it as another test case

To Reproduce

Create a test case with a before hook.
Execute the same test case twice : the first time it succeed. the second time it failed in the before hook

Expected behavior
The test case should be the same whenever it failed (in the before / before all / after / after all hook)

Environment (please complete the following information):

Cypress version: "^10.9.0"
cypress-allure-plugin version: "^2.37.0"

Oleksandr Shevtsov · Answer 1 · Wed Mar 29 2023 17:16:24 GMT+0800 (China Standard Time)

Thank you for opening an issue.

That ui is quite different from allure report, so I have no idea how to address it or verify.
Moreover "another test case" is identified by allure with historyId, which is a hash of the test name, so if test name was not changed - it is exactly the same execution result in allure, and will be linked to previous results if history is saved.

jeremie-val · Answer 2 · Wed Mar 29 2023 17:23:14 GMT+0800 (China Standard Time)

The interface you see it the one from Allure TestOps :)
nearly the same as allure report.

And we do not change the test name, the only difference in the 2 tests are the one which fail in the "before" hook, and the other one which succeed.

For example, the test which fail in the step is consider as the same as the succeed one.

Maybe it could come from the "full name" which are different (and visibile on the screenshot).
We have :
"before all" hook for "should update an existing client" and should update an existing client

jeremie-val · Answer 3 · Wed Mar 29 2023 17:33:28 GMT+0800 (China Standard Time)

Here are the 2 allure results generated from our tests :

The one with the hook which fail
The one with all tests in success
allure-results.zip

Sheth90 · Answer 4 · Mon Apr 03 2023 15:13:08 GMT+0800 (China Standard Time)

Hello team,
I have configured the allure report successfully with my setup. I used version conf as below.
Cypress: 12.5.1
After executing my test case. I am getting results for 2nd method, not for the first method.
TestBody of the first method is overridden by the second method's tc. Please help to resolve this issue with the allure report.

Oleksandr Shevtsov · Answer 5 · Mon Apr 03 2023 20:26:06 GMT+0800 (China Standard Time)

@Sheth90 your issue is not related to this one, as I do not see any failures of before hook, so I converted it to separate issue.
Further communication will be conducted in #201

Oleksandr Shevtsov · Answer 6 · Wed Apr 05 2023 09:32:14 GMT+0800 (China Standard Time)

Hi @jeremie-val
Issue is resolved in v2.38.3

Now plugin takes original test name for calculating history id hash.

jeremie-val · Answer 7 · Wed Apr 05 2023 20:08:01 GMT+0800 (China Standard Time)

Thank you very much, I'll have a look.

Sheth90 · Answer 8 · Thu Apr 06 2023 13:55:20 GMT+0800 (China Standard Time)

@jeremie-val @Shelex how to install or update allure report version to v2.38.3?

jeremie-val · Answer 9 · Thu Apr 06 2023 14:21:20 GMT+0800 (China Standard Time)

Just have to change the version in the package.json file.
That's what you did here : #201 (comment)

Just let @Shelex have a look :)

Sheth90 · Answer 10 · Thu Apr 06 2023 14:24:12 GMT+0800 (China Standard Time)

But still same problem persists. What to do for it? @jeremie-val As like #201

Oleksandr Shevtsov · Answer 11 · Thu Apr 06 2023 23:54:06 GMT+0800 (China Standard Time)

@Sheth90 you have a completely separate issue so I left a comment in #201

jeremie-val · Answer 12 · Fri Apr 07 2023 22:34:49 GMT+0800 (China Standard Time)

Validated today, a fail in a before hook is now integrated to the same test