zakame / earthly-kubernetes

PoC for running multi-node earthly-buildkitd on Kubernetes

earthly-kubernetes

This is a short proof-of-concept for deploying Earthly remote buildkit on Kubernetes, with the goals of exploring multi-node buildkit deployment (via DaemonSet to ensure each available node has a buildkit) and employing cert-manager to create a PKI for mTLS between buildkit and earthly clients.

Quick start

This repository is not meant to be cloned directly, unless for contributions.

You will need both kubectl and kustomize.

To get started with a basic workload with no mTLS configured, please do:

kustomize localize "https://github.com/zakame/earthly-kubernetes?ref=master" earthly-kubernetes
cd earthly-kubernetes
kubectl apply -k .

This deploys a DaemonSet of earthly-buildkitd for each node on the cluster, barring taints.

Once ready, test the DaemonSet by loading some jobs:

kubectl apply -f https://raw.githubusercontent.com/zakame/earthly-kubernetes/master/jobs.yaml

Then inspect logs for all jobs via:

kubectl logs -n earthly -f -l job=earthly

To add mTLS, first get the Kustomize component:

kustomize localize "https://github.com/zakame/earthly-kubernetes//mtls?ref=master" components

Then add this component and redeploy:

kustomize edit add component components/mtls
kubectl apply -k .

Then re-test with updated jobs:

kubectl delete jobs -n earthly -l job=earthly
kubectl apply -f https://raw.githubusercontent.com/zakame/earthly-kubernetes/master/mtls/jobs.yaml

Then reinspect with the same kubectl logs command as above.

Finally, for clean-up:

kubectl delete -k .