Define requirements for cloud config
andrewsykim opened this issue · comments
Each provider is doing something different with cloud config, and we don't have a consistent story for what we expect providers to do with it. We should have a discussion with the necessary stakeholders (cloud providers, api reviewers, etc) to better define what is expected for cloud config (future plans, depreciation policy, backwards compatibility, etc). Maybe the way things are today is fine (letting the provider define how cloud config is used), but we should at least document this expectation better.
/priority important-longterm
Some notes from discussion that was spawned by API reviewers:
- Config from cloud providers is a Kubernetes API (just like on disk config files or REST APIs or kubectl flags) and must have "rules"
- We don't have a lot of the rules formalized for cloud-provider, but we need to do that
- We need to ensure API review and backward compatibility guarantees are defined
- tests, versions, releases, how we ensure we don't regress, what things are allowed differently than other APIs
- We need sig-arch subproject for api review (@liggitt) to integrate / sign off on / help shape ^
@kubernetes/api-reviewers since I don't know that we talked about this much recently and I know I'm guilty of ignoring the cloud config stuff sometimes :(
Also, what is the intersection between conformance and config formats? Can two different distros of Kube have different cloud provider configuration files and both be conformant? Do we extend conformance to operational characteristics or not? @kubernetes/cncf-conformance-wg.
Triggered by in tree: https://github.com/kubernetes/kubernetes/blob/66674f549626cc41f04e475d2c0e865116c4cd40/pkg/cloudprovider/providers/vsphere/vsphere.go#L125 and assessing whether it has tests, is compatible, etc (tests eg https://github.com/kubernetes/cloud-provider-vsphere/blob/master/pkg/cloudprovider/vsphere/vsphere_test.go#L260)
Our v1.15 backlog is pretty packed, how important is it that we outline requirements for cloud config in this release? Can we punt this discussion to the next milestone or do you feel we should bump this into v1.15?
No cloud-provider features should be covered by conformance currently.
On config more generally: Conformance needs to assume that how a cluster is created is a black box, since there are many ways to provision and configure clusters.
Queued this discussion in for the next SIG call, please attend if you have opinions/thoughts on this
/assign @justinsb @frapposelli
Action item from SIG call: update docs to indicate that cloud config should follow the Kubernetes compatibility guarantees. If this doesn't fit into any of the existing docs, I think leaving inline comments in the in-tree cloud config structs would be sufficient.
For removing fields we need to give users a sufficient amount of time to update their cloud configs and ideally log warnings when the field is in use.