Define requirements for cloud config

Question

Define requirements for cloud config

andrewsykim opened this issue 5 years ago · comments

Each provider is doing something different with cloud config, and we don't have a consistent story for what we expect providers to do with it. We should have a discussion with the necessary stakeholders (cloud providers, api reviewers, etc) to better define what is expected for cloud config (future plans, depreciation policy, backwards compatibility, etc). Maybe the way things are today is fine (letting the provider define how cloud config is used), but we should at least document this expectation better.

Andrew Sy Kim · Answer 1 · Wed Mar 27 2019 23:38:18 GMT+0800 (China Standard Time)

/priority important-longterm

Clayton Coleman · Answer 2 · Wed Mar 27 2019 23:39:39 GMT+0800 (China Standard Time)

Some notes from discussion that was spawned by API reviewers:

Config from cloud providers is a Kubernetes API (just like on disk config files or REST APIs or kubectl flags) and must have "rules"
We don't have a lot of the rules formalized for cloud-provider, but we need to do that
We need to ensure API review and backward compatibility guarantees are defined
- tests, versions, releases, how we ensure we don't regress, what things are allowed differently than other APIs
We need sig-arch subproject for api review (@liggitt) to integrate / sign off on / help shape ^

@kubernetes/api-reviewers since I don't know that we talked about this much recently and I know I'm guilty of ignoring the cloud config stuff sometimes :(

Clayton Coleman · Answer 3 · Wed Mar 27 2019 23:40:59 GMT+0800 (China Standard Time)

Also, what is the intersection between conformance and config formats? Can two different distros of Kube have different cloud provider configuration files and both be conformant? Do we extend conformance to operational characteristics or not? @kubernetes/cncf-conformance-wg.

Clayton Coleman · Answer 4 · Wed Mar 27 2019 23:41:57 GMT+0800 (China Standard Time)

Triggered by in tree: https://github.com/kubernetes/kubernetes/blob/66674f549626cc41f04e475d2c0e865116c4cd40/pkg/cloudprovider/providers/vsphere/vsphere.go#L125 and assessing whether it has tests, is compatible, etc (tests eg https://github.com/kubernetes/cloud-provider-vsphere/blob/master/pkg/cloudprovider/vsphere/vsphere_test.go#L260)

Andrew Sy Kim · Answer 5 · Wed Mar 27 2019 23:48:21 GMT+0800 (China Standard Time)

Our v1.15 backlog is pretty packed, how important is it that we outline requirements for cloud config in this release? Can we punt this discussion to the next milestone or do you feel we should bump this into v1.15?

cc @mcrute @justinsb @dims @cheftako @frapposelli

Brian Grant · Answer 6 · Thu Mar 28 2019 01:13:23 GMT+0800 (China Standard Time)

No cloud-provider features should be covered by conformance currently.

On config more generally: Conformance needs to assume that how a cluster is created is a black box, since there are many ways to provision and configure clusters.

Andrew Sy Kim · Answer 7 · Fri Mar 29 2019 03:49:09 GMT+0800 (China Standard Time)

Queued this discussion in for the next SIG call, please attend if you have opinions/thoughts on this

Andrew Sy Kim · Answer 8 · Thu Apr 04 2019 04:34:25 GMT+0800 (China Standard Time)

/assign @justinsb @frapposelli

Andrew Sy Kim · Answer 9 · Thu Apr 04 2019 05:02:15 GMT+0800 (China Standard Time)

Action item from SIG call: update docs to indicate that cloud config should follow the Kubernetes compatibility guarantees. If this doesn't fit into any of the existing docs, I think leaving inline comments in the in-tree cloud config structs would be sufficient.

For removing fields we need to give users a sufficient amount of time to update their cloud configs and ideally log warnings when the field is in use.