Testing depends on alpha `sumtypes`
mtelka opened this issue · comments
Since commit 7ff3cee GitPython's tests started to depend on sumtypes which does not have any stable release available at PyPI and seems to be dead for more than two years now (see https://github.com/radix/sumtypes/commits/master/).
This causes inconvenience for downstream packagers where tests are run to make sure the constructed package is working properly. Could you please make the sumtypes test dependency optional (especially on non-Windows OSes)? Or switch off of it completely?
Thank you.
Thanks for pointing this out - I think this should be possible, but @EliahKagan would certainly know best.
Yes, the use of sumtypes can be eliminated altogether. I have opened #1798, which removes the test dependency on the sumtypes library.
While I do not think this is a reason to refrain from applying that change, I am unclear on the exact nature of the problem this currently causes for downstream maintainers. If that could be clarified, then I might be able to use that information to avoid introducing related problems for downstream maintainers in the future. I think that might also clarify whether a new release needs to be done for this change.
My thinking on this is detailed below. But to be clear, it is not intended as an argument against fixing this issue by merging #1798 (which I advocate doing), nor against further steps such as publishing a new release if that is needed. I think it is very valuable for downstream maintainers to be able to run the full test suite, and to be able to automate doing so without unnecessary impediments.
In general it is often reasonable to use code and techniques in testing that would not be reasonable to use in the code under test. Although the latest version of the sumtypes library still has an "a" suffix, it is available from PyPI and installable with pip. It is also not the only test dependency that has no stable release, in the sense most relevant to code quality.
For packages that use semantic versioning and have not had a 1.0.0 release, I believe the distinction between a release named in such a way that PyPI presents it as a prerelease (as sumtypes is, with the version suffix a6), and other pre-1.0.0 releases stated by their developers not to be stable, is often insignificant and cannot be discerned from the version numbers. Going by the classifiers in the package metadata, sumtypes was the only "alpha" test dependency, but not the only test dependency classified as not stable: the direct test dependencies pytest-instafail and pytest-sugar, as well as the indirect test dependency pathspec (through black), are all specified as "beta" by their classifiers.
However, it happens to be that those test dependencies can be omitted and it is still possible to run the tests (just not as they are run on GitPython's own CI, and not with the same experience as the readme documents). Furthermore, while the classifiers represent those packages as being in beta, beta is still typically more stable than alpha. The version numbers have no suffix that causes PyPI to show them as prereleases, but it seems to me that this is the least significant factor; however, I recognize that this judgment is subjective.
One reason I can think of that it would be undesirable to require sumtypes to be installed in downstream testing is if a distribution has the goal for all test dependencies to be available using only downstream packages (i.e., without accessing pip), for example so that they can be declared as downstream-package build dependencies. In that case, it would be inconvenient to handle cases where something needed as a test dependency would not otherwise be eligible for packaging under the downstream distribution's guidelines. It seems to me that avoiding such test dependencies will not always be an achievable or reasonable goal. However, even if that is the only rationale, in this case I think it may be sufficient, since the abstractions expressed with sumtypes can be expressed in another way, as detailed in #1798.
Thanks @EliahKagan. You expressed it very well. Our downstream OpenIndiana packaging and testing aims exactly for having all needed packages available natively, i.e. without a need to run pip. Also, we try to avoid to package versions that are marked as PRE-RELEASE by PyPI. That apparently includes a and rc versions. We skip such versions intentionally because our packing is (half-)automatic and it tries to do its job once it finds there was new version released at PyPI.
The sumtypes need for GitPython popped-up as a completely new dependency that would need to be newly packaged since no other package in our ecosystem (ca 600 Python packages) depends on it. That is usually doable but it still needs some resources to be invested. Not only for the initial packaging, but also to maintain it in future, for example to repackage it for new Python versions, etc.
When I looked closely at sumtypes and when I saw that the project is inactive for more than two years with alpha versions released at PyPI only then the immediate first thought was that it would be better to not package this project because it does not look viable at all. It is a maintenance nightmare when there is an inactive project that slowly rots and does not work well with newer versions of other projects or Python itself.
You mentioned pytest-instafail and pytest-sugar. Both are very good examples. We do not have them packaged (yet) because they are needed by GitPython only (IIRC) and the dependency is optional. So I opted to patch out both of them instead of packaging them.
Unfortunately, something like that is not so easily possible with sumtypes AFAIK. So for now I simply skipped affected tests using --ignore test/test_index.py. And I reported this bug. To either get better insights and assurance that sumtypes is worth packaging, or that you'll adjust GitPython so sumtypes dependency is optional or removed.
Since we already packaged 3.1.41 we do not need new release once #1798 is merged. Because, honestly, GitPython is not a critical part of our ecosystem. If I look properly the most important consumer is testing of pylint, so 3.1.41 packaged with test/test_index.py skipped is fully acceptable for us.
Thank you.
@EliahKagan: Hi! 👋
I package this project for Arch Linux. Adding an undead package as new dependency (even if for testing, because we do run tests) is a problem. Either I have to temporarily disable the test or package an undead dependency. :)
Since you now removed it, I'll likely just do the former, as I have way too many Python stuff to maintain already.
Either way, a new release clearing this up is always appreciated. Many thanks! 🎉
I apologize for not replying earlier! Although a specific post-release did not end up being done for this, all releases after this one should not have had the problem, as the dependency was removed in #1798. Specifically, 3.1.42, which came out a while ago, should not have had the problem, and the very recently released 3.1.43 should not either.
(3.1.43 causes typing-extensions to be a test dependency on more Python versions than before, because some new test modules import assert_type. I hope this is not a problem, and I also think it is not a problem because that library is ubiquitous, but please me know if that is not the case.)