estafette / estafette-gke-node-pool-shifter

Kubernetes controller that can shift nodes from one node pool to another, to favour for example preemptibles over regular VMs

Home Page:

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool


This controller shifts nodes from one node pool to another, in order to favour preemptibles over a 'safety net' node pool of regular vms.



You can either use environment variables or flags to configure the following settings:

Environment variable Flag Default Description
INTERVAL --interval (-i) 300 Time in second to wait between each shift check
KUBECONFIG --kubeconfig Provide the path to the kube config path, usually located in ~/.kube/config. For out of cluster execution
METRICS_LISTEN_ADDRESS --metrics-listen-address :9001 The address to listen on for Prometheus metrics requests
METRICS_PATH --metrics-path /metrics The path to listen for Prometheus metrics requests
NODE_POOL_FROM --node-pool-from Name of the node pool to shift from
NODE_POOL_FROM_MIN_NODE --node-pool-from-min-node 0 Minimum amount of node to keep on the from node pool
NODE_POOL_TO --node-pool-to Name of the node pool to shift to

Before deploying, you first need to create a service account via the GCloud dashboard with role set to Compute Instance Admin and Kubernetes Engine Admin. This key is going to be used to authenticate from the application to the GCloud API. See documentation.

Deploy with Helm

# Prepare Helm/Tiller
$ kubectl create sa tiller -n kube-system
$ helm init --service-account tiller
$ kubectl create clusterrolebinding tiller \
    --clusterrole=cluster-admin \

# Install
$ helm upgrade estafette-gke-node-pool-shifter \
    --namespace estafette \
    --install \
    --set rbac.create=true \
    --set-file googleServiceAccount=./google_service_account.json \
    --set nodePool.from=default-pool \
    --set \

Deploy without Helm

export NAMESPACE=estafette
export APP_NAME=estafette-gke-node-pool-shifter
export TEAM_NAME=tooling
export VERSION=1.0.11
export GO_PIPELINE_LABEL=1.0.11
export GOOGLE_SERVICE_ACCOUNT=$(cat google-service-account.json | base64)
export INTERVAL=300
export NODE_POOL_FROM=default-pool
export NODE_POOL_TO=preemptible-pool
export CPU_REQUEST=10m
export CPU_LIMIT=50m
export MEMORY_LIMIT=128Mi

# Setup RBAC
curl | envsubst | kubectl apply -n ${NAMESPACE} -f -

# Run application
curl | envsubst | kubectl apply -n ${NAMESPACE} -f -

Local development

For development purpose, you can create a new cluster with 2 autoscaled node pools, 1 preemptible and 1 regular VM.

Create the cluster with appropriate node pools

export CLUSTER_NAME=node-shifter
export CLUSTER_VERSION=1.7.3
export PROJECT=my-project
export ZONE=europe-west1-c

# Create cluster with regular VMs
gcloud beta container clusters create $CLUSTER_NAME \
  --project=$PROJECT \
  --zone=$ZONE \
  --cluster-version=$CLUSTER_VERSION \
  --num-nodes=1 \
  --enable-autoscaling \
  --min-nodes=0 \

# Add preemptible VMs node pool
gcloud beta container node-pools create preemptible-pool \
  --project=$PROJECT \
  --zone=$ZONE \
  --cluster=$CLUSTER_NAME \
  --num-nodes=1  \
  --enable-autoscaling \
  --min-nodes=1 \
  --max-nodes=3 \

Deploy an application

kubectl run nginx --image=nginx:alpine --replicas=5 --limits='cpu=200m,memory=512Mi'

Start the node pool shifter

# proxy master
kubectl proxy

# in another shell
go build && ./estafette-gke-node-pool-shifter --node-pool-from=default-pool --node-pool-to=preemptible-pool

Note: KUBECONFIG=~/.kube/config as environment variable can also be used if you don't want to use the kubectl proxy command.

If necessary, you can resize the node pool size:

gcloud container clusters resize $CLUSTER_NAME
  --project=$PROJECT \
  --zone=$ZONE \
  --size=1 \


Kubernetes controller that can shift nodes from one node pool to another, to favour for example preemptibles over regular VMs

License:MIT License


Language:Go 85.8%Language:HTML 13.2%Language:Dockerfile 1.0%