gitops-kyverno

Kubernetes' policy managed with Flux and Kyverno

Prerequisites

You will need a Kubernetes cluster version 1.21 or newer. For a quick local test, you can use Kubernetes kind. Any other Kubernetes setup will work as well though.

In order to follow the guide you'll need a GitHub account and a personal access token that can create repositories (check all permissions under repo).

Install the Flux CLI on MacOS or Linux using Homebrew:

brew install fluxcd/tap/flux

Repository structure

The Git repository contains the following top directories:

apps dir contains a demo app (podinfo) and its configuration for each environment
infrastructure dir contains common infra tools such as Kyverno and its cluster policies
clusters dir contains the Flux configuration per cluster

├── apps
│   ├── base
│   ├── production 
│   └── staging
├── infrastructure
│   ├── configs
│   └── controllers
└── clusters
    ├── production
    └── staging

Bootstrap the staging cluster

Create a cluster named staging with kind:

kind create cluster --name staging

Fork this repository on your personal GitHub account and export your GitHub access token, username and repo name:

export GITHUB_TOKEN=<your-token>
export GITHUB_USER=<your-username>

Set the kubectl context to your staging cluster and bootstrap Flux:

flux bootstrap github \
    --context=kind-staging \
    --owner=${GITHUB_USER} \
    --repository=gitops-kyverno \
    --branch=main \
    --personal \
    --path=clusters/staging

The bootstrap command commits the manifests for the Flux components in clusters/staging/flux-system dir and creates a deploy key with read-only access on GitHub, so it can pull changes inside the cluster.

Wait for Flux to install the controllers and the demo app with:

watch flux get kustomizations

Access the Flux UI

To access the Flux UI on a cluster, first start port forwarding with:

kubectl -n flux-system port-forward svc/weave-gitops 9001:9001

Navigate to http://localhost:9001 and login using the username admin and the password flux.

Weave GitOps provides insights into your application deployments, and makes continuous delivery with Flux easier to adopt and scale across your teams. The GUI provides a guided experience to build understanding and simplify getting started for new users; they can easily discover the relationship between Flux objects and navigate to deeper levels of information as required.

Mutating deployments with Kyverno

In the infrastructure/configs dir there are two Kyverno policies that mutate Kubernetes Deployments to set a restricted security context and to replace the app container image tag with its digest.

Even if in Git, the podinfo image is set to ghcr.io/stefanprodan/podinfo:6.2.3, the actual deployment image in-cluster is mutated by Kyverno and the tag is replaced with the image digest. Same thing with the security context, in Git, podinfo has no such fields, but in-cluster the deployment ends up with:

$ kubectl -n podinfo get deployments.apps podinfo -oyaml | yq '.spec.template.spec.containers[0]'
image: ghcr.io/stefanprodan/podinfo@sha256:4a72d3ce7eda670b78baadd8995384db29483dfc76e12f81a24e1fc1256c0a8e
imagePullPolicy: IfNotPresent
name: podinfo
ports:
  - containerPort: 9898
    name: http
    protocol: TCP
securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
      - ALL
  readOnlyRootFilesystem: true
  runAsNonRoot: true
  seccompProfile:
    type: RuntimeDefault

Flux vs Argo drift detection

If you've been using Argo, and you're switching to Flux, you may expect for Flux to report the above changes as a drift from Git and reapply the deployment continuously. With Argo, you must define policies for each field that Kyverno mutates to ignore differences, while with Flux this is not need. Flux works with any admission controller as long the mutation also happens at dry-run.

To test that Flux has included the Kyverno mutations in its "desired state" we can run flux diff and check that no differences are reported:

 flux diff kustomization apps --path ./apps/staging/

Now if we change the podinfo image tag in apps/staging/kustomization.yaml to 6.3.0 and we run diff:

$ flux diff kustomization apps --path ./apps/staging/
✓  Kustomization diffing...
► Deployment/podinfo/podinfo drifted

metadata.generation
  ± value change
    - 2
    + 3

spec.template.spec.containers.podinfo.image
  ± value change
    - ghcr.io/stefanprodan/podinfo@sha256:4a72d3ce7eda670b78baadd8995384db29483dfc76e12f81a24e1fc1256c0a8e
    + ghcr.io/stefanprodan/podinfo@sha256:c0d72aa8829a310b998308b8d3f05bde6840c66eaf2862b218f87e21ea5fb275

⚠️ identified at least one change, exiting with non-zero exit code

We see that Flux detects a drift in the digest, instead of the image tag. Unlike Argo and other tools, Flux uses Kubernetes server-side apply dry-run to trigger the mutation webhooks, before it runs the drift detection algorithm.

With Flux, you don't have to create special rules for each mutation made by admission controllers.

stefanprodan / gitops-kyverno