Run acceptance tests in parallel
olevski opened this issue · comments
Is it possible to run the acceptance tests against a real renku deployment but from within a github action and in parallel?
This would allow us to do a few things:
- retry just a smaller set of tests (or just even a single test) when there is a failure or flaky test
- get our acceptance tests to run quicker
- get a nicer view of what failed and easier access to logs (i.e. we can have the test logs as github ci artefacts and not have to upload them to a s3 bucket)
But I am not sure if this is possible with how the acceptance tests are setup / written.
In addition to this if we can run the acceptance tests outside of helm like this we can then also slim down our helm tests to contain just a very small subset of the full suite of acceptance tests. Helm tests are supposed to be quick anyhow. And the fact that ours take a long time makes it really messy and confusing for flux. I.e. flux expects that a helm chart to be done deploying very quickly (and this includes the tests). But ours take almost 2 hours.
I think, I'm not entirely sure if I understand how do you see that Tasko:
- are you thinking about having each scenario running as a separate GH action?
- or just parallelizing them within a single GH action (that's rather simple and I think we used to experiment with that a bit).
By the way, we used to have them running as a GH action some time ago and then we decided to move it to helm test but I don't remember the reasons why.