Can I have the action fail if tests fail?
BradLucky opened this issue · comments
I have noticed that teeing the output of pytest in order for pytest-coverage-comment to leave a comment means that test failures do not cause the action to show as failed in Github. In fact, it hasn't "failed," since the exit code is swallowed by teeing the output.
I have found two alternatives to address this situation:
- Don't tee the output and simply use an XML report as described in the docs. The downside to this is that a comment is not left on the PR.
- Add another step after the commenter testing if there were any errors or failures and return an exit code if so. This adds a step to the workflow, but leaves a comment and still reports the action as failed.
Is there a way built in to pytest-coverage-comment to cause the action to fail if there are failures? If not, I feel it would be a worthwhile option. In fact, it's the expected behavior IMHO. I did not know when implementing this commenter that my PRs would suddenly "go green" even when tests failed.
I was just looking for this too! Looks like it was discussed in #69 and workarounds given there. I'll try those, though I do think the intuitive thing is an option that defaults to True for failing the step on test failure.
teeing the output is like a simple "workaround" to get the comment-report without modifying the workflow too much.
The best thing to do is to use the pytest-xml-coverage-path (--cov-report "xml:coverage.xml), in this way you will get the report-comment and fail the action when the pytest will fail or the solutions that provided #69