How to handle retries?
sobrinho opened this issue · comments
Hi there!
Our app have some flaky specs and we are retrying using --only-failures to retry just the flaky specs, which in general just works.
We are generating the junit.xml in the first try and then if a spec fails, we run the failed specs again a second time without it but this creates an issue where the CI interfaces says that the build didn't pass (because there are failed specs in the XML) but actually it did pass.
When that happens, we need to look at the build exit status to confirm if it failed or not.
What you think about updating/integrating with the --only-failures to update the previous XML somehow to make the interface more reliable?
PS: We can't use rspec-retry because we to do some cleanup between the retries and also looks more reliable to use the native --only-failures than a 3rd gem.
This is intentionally a pretty simple formatter and doesn't attempt to read or compose with existing results.
I'd suggest creating separate output files and merging them together afterwards.