proof-of-concept for running kustomize krm functions in kubernetes, potentially from within kubernetes.
Using a GitHub autogenerated name until I can give this a proper "short" name.
Status: Unfinished, but completed for now, see Results.
Kustomize currently can run containerized KRM functions only through docker run.
But this does not work well with GitOps tools like ArgoCD, which run on kubernetes, where docker run will not work.
An alternative can be found in this ArgoCD issue regarding this problem. My understanding is that relies on running nested containers using a privileged podman container. This might work, but I would like to research how this can be done without escalated privileges and be delegate to the platform.
I would like fo research and prove that running KRM Functions as Kuberneted Pods is possible with the official client libraries, specifically client-go.
- Base:
- Passing stdin
- Capturing stdout/stderr
- Parsing
docker run- Level 1: image & environment variables
- Level 2: mounts
- Level 3: networking restrictions
Status: Incomplete, but finished.
A go library that can run KRM functions in Kubernetes, through official client libraries.
The library will use the standard client library means to schedule the Pod
and use remotecommand functionality to provide the ResourceList input and capture the result.
Partially implemented:
- 👎 improper pod readiness check (currently simply 1s sleep)
- 👍 Passing stdin works fine
- 👍 Capturing stdout works fine
- 👎 We can not reliably capture stderr form the very start of the function because some time passes between the pod is ready and we are attached to it.
- 👎 When the pods are scheduled all function outputs will be written to the logging aggregator
A a (go) binary that can be used to replace the docker binary Kustomize uses for docker run, instead running a Kubernetes Pod using the above go library.
Status: Not started.
Passing ENV vars should be easily doable via pod spec.
Status: Not started.
mapping local fs contents using configmaps? what about fs structures that are not flat?
using something like kubectl cp?
seems to rely on tar, requiring that would be questionable.
could we use our own tar container as an init container? are init containers supported by remotecommand, can they have stdin?
Status: Not stared.
The central points, providing stdin and capturing stdout, are possible via remote client.
Unfortunately stdout will also be captured by the logging driver and, for example in OpenShift, persisted to log store. That is quite unfortunate as for example the generator I had mind produces a quite huge manifest, and if that manifest than is also run through containerized transformers, the log store will explode!
As such, the PoC will stop for now and remain in it's current state.