Failed test runs show as green in Slack
you1anna opened this issue · comments
Hi @YOU54F
On v1.1.8-alpha I am seeing failed runs resulting in green slack posts, as if they have passed.
We are only using SLACK_WEBHOOK_URL but I have seen this working previously, reflecting passed/failed runs correctly.
Is there a need to also supply SLACK_WEBHOOK_FAILED_URL to handle failed runs?
Thanks
You should be able to provide either SLACK_WEBHOOK_URL or SLACK_WEBHOOK_FAILED_URL to send failed alerts to a specific channel.
It should only returned a status of passed, if totalFailures === 0
Might be worth checking the test report json, that the logic is being used against
I can see value in debug mode, logging out the report contents to the command line,
It's hard to debug this in CI because we don't control it. It would be good to understand how you are debugging this locally.
Are you just running script.ts with breakpoints, or can you suggest a better way? Thanks!
Just from looking at the code, and the points at which it determines the test status.
There are tests to cover the report status, that help locally. it would only return a pass if there is no failures in the test report.
I’ve got an free slack instance, os circleci account and a couple of projects that consume this that I use for remote testing, but I don’t use the custom url option, that was a feature request that I haven’t personally tested e2e.
One of the easiest ways to diagnose the test reports and artefact location is to ssh into the ci box if your ci provider supports that. I can do that on circleci to navigate around the box, check the various artefacts, cat the test report and check the contents.
Just read that you can’t control ci, yeah that’s a pain, I have the liberty of full access in non prod envs at work, but an open source account on circleci costs me nowt and was the easiest way for me to play around on this!
You could always add something into your ci script, a step to cat the test report and ls some of the artefact dirs, and store them for inspection later.
Hi @YOU54F we are still seeing instances of this happening in CI. I have an example of mochawesome.json that I could send to you, but would rather not share it here. You can see from the screenshot the build returns green but there is a failed.png file attached.
Hey @YOU54F I've made the gist but unable to see the email you posted (*'d out). How can we solve this?