Update testing strategies for CI performance and robustness
bollwyvl opened this issue · comments
Elevator Pitch
Run the tests in parallel with pytest-xdist
.
Motivation
In addition to being faster, pytest-xdist
can find situations where file-based things (e.g. *.pyc
) can create issues when run concurrently, which do occur in the wild. It works fine with pytest-cov
, as well.
To make this work, the tests that use the ref
and package
ifxtures in test_importnb
need some work to be more hygienic.
Another CI test speed thing is to skip coverage on pypy: it's a lot slower, and can cause spurious issues (especially on windows) that wouldn't impact real-world behavior where speed is the only goal.
In CI that would just be a conditional step based on python-version
, potentially moving some of the configuration arguments out of the pytest options, and into the specific invocation in the hatch jobs.