dgzlopes / termination-handler

Handle termination notices on spot/preemptible instances.

termination-handler

About

termination-handler handles termination notices on spot/preemptible instances.

As an example, if deployed on a Kubernetes cluster and a termination notice is issued by the cloud provider, termination-handler drains the node it is running on before the node is taken away by the cloud provider.

Inspired by pusher/k8s-spot-termination-handler for AWS, termination-handler surges from the need to operate the same tooling in various distinct environments (providers, orchestrators...)

Features

• Supports multiple cloud providers (AWS, GCP).
• Supports multiple handlers (Kubernetes, Nomad, Slack).
• Small and extensible.

Documentation

TBD

Usage

Deploy to Kubernetes

A K8s docker image is available at dgzlopes/termination-handler-k8s and sample Kubernetes manifests are available in the deploy/k8s folder.

To deploy in clusters using RBAC, please apply all of the manifests (Daemonset, ClusterRole, ClusterRoleBinding and ServiceAccount) in the deploy/k8s folder but uncomment the serviceAccountName in the Daemonset.

Requirements

For termination-handler to schedule correctly; you will need an identifying label on your spot/preemptible instances.

We add a label node-role.kubernetes.io/spot-worker to our spot/preemptible instances and hence this is the default value in the node selector of the Daemonset.

nodeSelector:
  "node-role.kubernetes.io/spot-worker": "true"

To achieve this, add the following flag to your Kubelet:

--node-labels="node-role.kubernetes.io/spot-worker=true"

Configuration

To define any custom parameters to the drain command you can use DRAIN_PARAMETERS environment property. If not defined, default parameters are --grace-period=120 --force --ignore-daemonsets.

env:
  - name: DRAIN_PARAMETERS
    value: '--grace-period=120 --force --ignore-daemonsets --delete-local-data'

Deploy to Nomad

A Nomad docker image is available at dgzlopes/termination-handler-nomad.

TBD

Demo mode

The main way to use termination-handler is waiting for the termination notice from the cloud provider. However, termination-handler comes with a demo mode that is can simulate the notice. When deployed it will identify your cloud provider and run your handlers.

To activate termination-handler demo mode on Kubernetes, you can use DEMO_TERMINATION_HANDLER environment property.

env:
  - name: DEMO_TERMINATION_HANDLER
    value: True

Other handlers

Slack

Sends a notification message to a specific channel when a termination notice is issued.

How to contribute

1. Check for open issues or open a fresh issue to start a discussion around a feature idea or a bug.
2. Fork the repository on GitHub to start making your changes to the master branch (or branch off of it).
3. Write a test which shows that the bug was fixed or that the feature works as expected.
4. Send a pull request and bug me until it gets merged and published.

Some things that would be great to have:

• Support for Execution handler (certain command or task)
• Support for notifications (Datadog..)
• Option to omit selected cloud provider discovery.