Gardener-GKE

Disclaimer: This project sets up a Gardener landscape on a GKE cluster. This is by no means a productive setup!

Overview

Prerequisites

Kubernetes cluster (min 4x n1-standard-2 nodes)
Domain/Zone in CloudDNS
GCP serviceaccount
terraform
yaml2json
jq
cfssl
openssl
kubectl
helm

There's a docker image with the dependencies available at afritzler/virtual-gardener-gke

Setup

Prepare setup.yaml

First we need to clone and configure our setup

git clone https://github.com/afritzler/virtual-gardener-gke.git
cd virtual-gardener-gke
cp setup.yaml.example setup.yaml

Create GKE cluster

export GKE_CLUSTER_NAME=gardener
export GCP_PROJECT=`gcloud config get-value project`

gcloud container clusters create $GKE_CLUSTER_NAME --num-nodes=4 --machine-type=n1-standard-4 --zone=europe-west1-b --enable-basic-auth --password f00bar

You will need a kubeconfig with basic-auth user authentication:

export KUBECONFIG=/tmp/kubeconfig
gcloud container clusters get-credentials $GKE_CLUSTER_NAME --zone europe-west1-b --project $GCP_PROJECT
mv kubeconfig $KUBECONFIG
src/bin/convertkubeconfig

The basic authentication credentials for your cluster can be found under cluster details -> "Show credentials".

You need to enter admin/f00bar (from above) and it should be created at ./kubeconfig

Create GCP serviceaccount

gcloud iam service-accounts create gardener --display-name "Gardener"
gcloud projects add-iam-policy-binding $GCP_PROJECT --member="serviceAccount:gardener@$GCP_PROJECT.iam.gserviceaccount.com" --role="roles/editor"
gcloud iam service-accounts keys create ./google-serviceaccount.json --iam-account gardener@$GCP_PROJECT.iam.gserviceaccount.com

Then, edit the setup.yaml accordingly (e.g. paste the contents of google-serviceaccount.json)

Local dependencies

Either install dependencies locally or use the provided docker image:

docker run --rm -it -v $(pwd):/gardener -w /gardener afritzler/virtual-gardener-gke

Deploy Ingress Controller + Ingress DNS Record

src/ingress-controller/deploy

Deploy Etcd

Deploy the Etcd needed by the Gardener extension API server

src/etcd/deploy

Deploy Identity

src/identity/deploy

Deploy Virtual Kube-Apiserver

src/virtualapiserver/deploy

Deploy Gardener

Deploy the Gardener extension API server and controller

src/gardener/deploy

Configure the Gardener Landscape

Deploy and configure CloudProfile, Seed, etc ...

src/gardenconfig/deploy

Deploy the Gardener Dashboard

src/dashboard/deploy

Accessing the Gardener Dashboard

To access the Gardener Dashboard, use

cat state/identity/dashboardurl

to figure out the dashboard URL.

Interacting with the Virtual Setup

The kubeconfig can be found under state/virtualapiserver/kubeconfig.yaml. So in order to deploy something to the Gardener API server you need to run

kubectl --kubeconfig=state/virtualapiserver/kubeconfig.yaml apply -f examples/shoot.yaml

Cleanup

Remove Shoot

Delete the created shoot cluster (a simple kubectl delete shoot NAME is not allowed in order to prevent users from accidentally deleting their clusters – instead, they need to confirm upfront that the deletion is fine by annotating the shoot resource. You can use this script to do that: https://github.com/gardener/gardener/blob/master/hack/delete (./hack/delete shoot gcp-test garden-core)).

To do it the manual way

kubectl --kubeconfig=state/virtualapiserver/kubeconfig.yaml -n garden-core annotate shoot gcp-test confirmation.garden.sapcloud.io/deletion=true --overwrite
kubectl --kubeconfig=state/virtualapiserver/kubeconfig.yaml -n garden-core delete shoot gcp-test

Remove Gardener Config

kubectl --kubeconfig state/virtualapiserver/kubeconfig.yaml annotate project core confirmation.garden.sapcloud.io/deletion=true --overwrite
kubectl --kubeconfig state/virtualapiserver/kubeconfig.yaml delete -f gen/gardenconfig/config.yaml

Remove Gardener

helm delete --purge gardener

Remove Virtual API Server

helm delete --purge virtual-apiserver

Remove Identity

helm delete --purge identity

Remove Gardener Dashboard

helm delete --purge gardener-dashboard

Remove Etcd

helm delete --purge virtual-garden-etcd

Remove Ingress Controller + DNS Record

helm delete --purge nginx-ingress-controller
# to delete the DNS record
./src/ingress-controller/destroy

Remove the Garden Namespace

kubectl delete ns garden

afritzler / virtual-gardener-gke