Improve testsuite performance (speed, parallelization) [33]

[Jochen Breuer]

Salt TestSuite Documentation: https://docs.saltstack.com/en/latest/topics/development/tests/index.html#salt-s-test-suite

Checkout: saltstack/salt#53566

Speed improvements

• identify slow tests (use profiler)
• Investigate the reason of the slowest tests: #95
• create PR to improve performance

Parallelization

• research how to group the tests in relevant units
• create PoC of running the groups in parallel
• present results

[Wayne Werner]

Here's the slowest tests - about 1,000 - from Salt (pulled from Jenkins).

SaltStack tests that never ran _1s - Tests to skip (_1s).csv.txt

For each of the platforms that we test (see Jenkins), these are the tests that never ran <1s. Every single test took >=1s to run. I included the max and min test times, since some of the tests ran faster on one platform than the other. The times are in seconds.

This represents about 10% of the tests that are executed on any platform.

For some extra flavor, here's another breakdown by test times:

The legend on the right is the threshold in seconds. About 75% of tests take <0.01s, and then about another ~7% of tests <0.1s, <1s respectively, to get ~90% of tests <1s.