Automated tests needed for installation
nadenf opened this issue · comments
Given the number of regressions with installation it would be useful to have a matrix suite of automated tests covering:
- Inventory types e.g. EKS, OKD
- Install methods e.g. Ansible Playbook, Podman
- Operating systems e.g. Centos, Ubuntu
- Users e.g. root, non-root
It would be too costly to have run on every PR but perhaps run daily or weekly.
Hi @nadenf, we have a CI checking a subset of all the possible scenarios, the periodic jobs run daily and we test each distro + having centos in the HV, also we test a combination of 1-3 controllers and 0-3 compute nodes is what we show here https://github.com/Kubeinit/kubeinit/blob/main/kubeinit/README.md
But yeah there are many scenarios not covered by the CI.
One option is to ditch whatever is not in the CI (like allowing Ubuntu/Debian in the HVs), but I wanted to allow people running the playbooks if they have Ubuntu as the host.
I think this can be something valuable as a discussion to see what we will like to support in the CI and if it's valuable to validate that what is not in the CI should not be deployed.
One of the reasons you might find some regressions is that we did a major refactor to allow things like Retry a failed deployment and how having the need to redeploy from scratch, we also added by default OVN as the overlay network.
Once we get to a point where all these nits are fixed, we should cut a new release with all these major changes.
If you have ideas about how to present this CI matrix feel free to jump here and give your thoughts, if you see a pattern in other projects we can try to follow them.
@gmarcy WDYT?
Something that may help here is to simplify the supported path:
i.e. only support and show Podman installation running as root in the README
And other setups e.g. Ansible Playbook just leave in the docs.
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days