Many tests fail on Windows
felix-seifert opened this issue · comments
If we look at the Run tests step on Windows agents, we can see that most of the tests fail. Interestingly, the step still passes despite those failures.
As the tests show ERROR, the issue should be within the fixtures of a test. For tests in the class TestGh, the errors occur when calling the teardown fixture shutil.rmtree. If I eliminate the call to shutil.rmtree by simply setting GH_KEEP_TMP, these tests do not fail on any OS.
My question is therefore whether we need this teardown. And if so, do we have an alternative? Alternatively, does anyone know which relevant part of Windows is unique and how we can fix this?
As a simple workaround, we can obviously stop testing on Windows. However, even though many programmers might be using Linux or Mac and do not require Windows, I am strongly against excluding the developers actually using Windows.
The errors due to the failed file deletion in the test teardown are fixed now. Nevertheless, some tests still fail on Windows complaining about a wrong assertion.
In PR #187, the above-mentioned issue of having failures on Windows machines with all Python versions is fixed. You can just investigate the GH Actions steps.
To address the issue of failing tests not letting a whole job fail, I have raised issue #201.