remote config via configmap and secrets
cgwalters opened this issue · comments
This overlaps with #7 some.
Here, the basic idea is something like:
bootc config add [--root=/etc] https://examplecorp.com/config.yml
(OR with support for OCI Artifacts we support bootc config add [--root=/etc] registry:quay.io/examplecorp/config-server-base:latest
)
Where config.yml
is a standard Kubernetes ConfigMap. By default, we "mount" the keys to /etc
. Then, bootc upgrade
looks for updates to all provided configmaps - if any change, it triggers the same upgrade logic as the base image.
We also fetch and handle secret objects in the same way. It'd be cool though to support something like handling encrypted secrets (and configmaps) which need to be decrypted via a key (which could be in a TPM or so).
We also need to think carefully about file permissions; mode 0644 for all configmap files and 0600 for secrets may make sense. In addition we could support special annotations to override these.
(This should also work to be invoked immediately after bootc install
to have it ready on the first boot, i.e. we also have a --root
argument or so)
To flesh this out a bit more, bootc upgrade
would check for updates to these things too.
Also, another variant of this is that we should support fetching all this stuff from the cloud user data too.
So this design would work in an obvious way outside of Kubernetes - we're using a standard API type, but we're not depending on an API server or kubelet, etc.
But, in a Kubernetes context, it would make total sense to fetch this data from the API server instead of just running http. (And potentially use proper watches to react to events instead of polling). I'm thinking we probably want to make the mechanism to fetch this data pluggable; ultimately it might be that there's /var/lib/bootc/config.d
and a privileged container image could e.g. use kubelet's credentials to fetch from the API server instead, and just hand things down to bootc.
This intersects with openshift/machine-config-operator#3327 too.
This intersects with openshift/machine-config-operator#3327 too.
A hugely interesting topic here of course is whether we should have a declarative way to set this stuff up. Today for example we have an imperative interface in bootc switch
to specify the root image.
But...following the kubelet "static pod" model, and building on the pod analogy above, perhaps we have
/etc/bootc/host.yaml
which is a subset of the pod spec, and allows declaratively writing the attached configmaps.
(This topic also relates to coreos/rpm-ostree#2326 )
It may also be interesting here to try to support something like an annotation which says the configmap changes should be live-applied by default - in the general case we'd probably need some sort of "post-apply hook" to ask affected services to reload. Or, such services could use inotify of course (at the risk of seeing partial states).
To implement this I'm thinking:
- Add a
kubernetes
feature (on by default) - Use the https://docs.rs/k8s-openapi/latest/k8s_openapi/ crate
- Support basic fetch via http, and investigate OCI artifact support (going to need to be driven into skopeo most likely)
- Write configmaps as ostree commits (map name to ref, etc.); although we may want to actually split into "metadata" and "contents"; metadata changes to the configmap shouldn't appear as changes probably, except things like the potential update semantics
- Create merged commit by applying configmap commits into
/usr/etc
to start - and here we have the interesting question of handling conflicts (i.e. what happens if my configmap has/etc/sudoers
and the host has that file too? I'd probably vote that we error by default and require an annotation to overwrite?) bootc update
looks for changes to base and all configmaps
Now...what many people will want is a read-only /etc
; if we have that, it becomes logically easier to have each configmap be a mounted tmpfs
or whatever, which we can more easily implement individual live-apply semantics.
Trying to play around with this, the ergonomics are definitely somewhat annoying because configmaps can't have /
in keys. If one wanted to e.g. write drop a a CA certificate into /etc/pki/ca-trust/source/anchors
and also a local registry mirror config in /etc/containers/registries.conf.d
it'd require separate configmaps with separate mountpoints. Maybe we just accept that for now.
Edit: Or, we could accept a List
object of configmaps.