Bolster performance of test suite execution
iAndriy opened this issue · comments
Is there an existing issue for this?
- I have searched the existing issues
Is your feature request related to a problem? Please describe.
As software engineer I would like to have fast test execution cycle, so the jobs associated with CI pipeline complete faster, hence reducing resources usage time.
Current execution of tests takes 7+ minutes (though it depends on retries), ideally I would like it to take few minutes and going under ten minutes would be a good start.
Describe the solution you'd like
I would like to use pytest-xdist to parallelize test execution.
Additional context
Quite many tests run the server on some hardcoded port, so a bit of care might be needed to enable parallel tests.
Quite many tests run the server on some hardcoded port, so a bit of care might be needed to enable parallel tests.
this makes sense, sounds like interesting problem to overcome, perhaps we can ask for the available port from os 🤔
An excellent goal, but that is going to be a painful task. As mentioned, because we're doing a lot of io, there are a lot of tests that are much more integration than unit. There's a lot of baked in synchronization and state management. It will be a huge effort to overcome and im not sure we'd be able to cover all of the same use cases. Port selection alone isn't the only obstacle.
Perhaps a more realistic goal may be to separate the test suites into a few subgroups and run each of those simultaneously. But not sure that'd be much better as it would explode the number of runners.
Installing dependencies is also a big slowdown with the current CI. Perhaps that can be sped up somehow? Using a Docker image with everything preinstalled, for instance?
That's a great idea. I'm all for this. We did this at work and it was a huge time saver.
@ahopkins @Tronic Perhaps Can you please share some links with examples ?
From my findings the majority of time is spend on running tests.
Here is example with annotations on screenshots. As you can see job took 4mins 12 secs( 252 seconds), within this time tests has been executed in 245.45 seconds.
@iAndriy This is correct, the vast majority of time is consumed in pytest. Different platforms are run in parallel but it still takes long, and this is cumbersome when running them locally, too.
Many tests use sleep and/or timeouts that would greatly benefit of parallel execution (or of mock clock). There are parts where unit tests rather than test client could be used, still ensuring coverage and desired tests, but making it faster, while currently the vast majority of tests are integration tests though the test client. There surely is also unnecessary overlap with different tests, and some might not even be relevant anymore.
But going through them to optimize and improve, making sure not to miss anything important, is a huge task that I doubt there is manpower to. Test parallelization might yield some easy results but it still needs care not to make the tests fail at random. Also looking at the few particularly slow tests in detail might yield useful speedup. As would reducing the number of combinations in parametrized tests. And reducing sleep/timeout durations (as long as no random failures occur then).
@iAndriy This is correct, the vast majority of time is consumed in pytest. Different platforms are run in parallel but it still takes long, and this is cumbersome when running them locally, too.
Many tests use
sleepand/or timeouts that would greatly benefit of parallel execution (or of mock clock). There are parts where unit tests rather than test client could be used, still ensuring coverage and desired tests, but making it faster, while currently the vast majority of tests are integration tests though the test client. There surely is also unnecessary overlap with different tests, and some might not even be relevant anymore.But going through them to optimize and improve, making sure not to miss anything important, is a huge task that I doubt there is manpower to. Test parallelization might yield some easy results but it still needs care not to make the tests fail at random. Also looking at the few particularly slow tests in detail might yield useful speedup. As would reducing the number of combinations in parametrized tests. And reducing sleep/timeout durations (as long as no random failures occur then).
@Tronic Thanks for elaborating, a lot of useful details I haven't know about.
yeah, it's important to keep tests deterministic, flakiness is a huge problem and being conscious of it during optimisation is important. As per your earlier suggestion re-organizing tests and adding dynamic ports to enable parallelism shall do the work. Surprisingly, some of the tests fail on my local machine but work during CI/CD, so maybe adding some utilities to run tests in docker (on local machine) would be useful too.
I intend to focus more on the dynamic port allocation and addressing sleeps, once we have this fully added start with the xdist/re-org of tests into specific folders.
I found quite many tests failing both on local MacOS and some versions of Linux. These were addressed in #2844 in case you want to dig up the diff or specific commits from that branch. With those changes I got the tests passing on MacOS and various Linux distros, not sure if I tried Windows though. The current main still fails many tests on my Macbook.