GKE Node Drainer
A tool to drain GKE preemptible nodes
Inspired by https://github.com/estafette/estafette-gke-preemptible-killer, this tool drains the nodes that arent covered by estafette-gke-preemptible-killer.
Why
Nodes sometimes slip from estafette-gke-preemptible-killer and get terminated before estafette-gke-preemptible-killer gets to rotate it. This tool makes sure that if a preemptible node gets deleted, it gets cordoned and drained to ensure a graceful shutdown is achieved.
How
Google logs on stackdriver whenever an instance is getting removed from an instance group, which needs to be exported to pub/sub. Once a message is sent to the topic, we consume it, then cordon the node and drain the pods on it. This is to make sure pods are gracefully terminated.
Usage
1. Create Pub/Sub Topic & Subscription
gcloud pubsub topics create --project <project> <topic_name>
gcloud pubsub subscriptions create <subscription_name> --topic=<topic_name> --project <project>
2. Create stackdriver export sink
Create the sink to export logs to and grant the service account generated in the command above Pub/Sub Publisher role on the topic you created.
$ gcloud logging sinks create <sink> pubsub.googleapis.com/projects/<project>/topics/<topic> \
--log-filter="protoPayload.methodName='v1.compute.instanceGroups.removeInstances' \
protoPayload.serviceName='compute.googleapis.com' operation.producer='type.googleapis.com'"\
--project <project>
--- output ----
Created [https://logging.googleapis.com/v2/projects/<project>/sinks/<sink>].
Please remember to grant `serviceAccount:<service_account>` the Pub/Sub Publisher role on the topic.
3. Create service account
Create a service account that will be used by the tool to subscribe to the topic
export project=<project>
gcloud iam --project=$project service-accounts create node-drainer \
--display-name node-drainer
export SERVICE_ACCOUNT=$(gcloud iam --project=$project service-accounts list --filter node-drainer --format 'value([email])')
gcloud projects add-iam-policy-binding $project \
--member=serviceAccount:$SERVICE_ACCOUNT \
--role=roles/pubsub.subscriber
gcloud iam --project=$project service-accounts keys create \
--iam-account $SERVICE_ACCOUNT \
google_service_account.json
Environment Variables
| Environment Variable | Description | Default |
|---|---|---|
| PUBSUB_SUBSCRIPTION | subscription to listen to | " " |
| KUBECONFIG | kubeconfig file if using out of cluster auth | " " |
| KUBERNETES_SERVICE_HOST | k8s host to connect to (within cluster auth) | Supplied by k8s by default |
| KUBERNETES_SERVICE_PORT | k8s port to connect to (within cluster auth) | Supplied by k8s by default |
| KUBERNETES_NAMESPACE | k8s namespace to use | Supplied by k8s by default |
| INSTANCE_GROUPS | (comma seperated list) gke instance groups to check if node belongs to | " " |
Build
docker build . -t gke-node-drainer
Deploy with Helm
helm install -n autoscaler helm/gke-node-drainer