deadman-switch

This tool implements a deadman switch for software systems. It expects to regularly receive pings from its configured services and one or more webhooks will be called when it doesn't hear from a service for a configurable amount of time.

Features

alert you when your services are down
alert you when your services up again
notifications can be send to any webhook or to slack
- use custom URL, headers, body for webhooks
- use custom key/value pairs on the slack message
configurable message debouncing
dynamic configuration of services and notifications via HTTP API
- secured with basic auth
scalable in both directions
- from a small container with <32MB RAM
- to a cluster that can handle thousands of pings and notifications per second
leader election in the cluster, so only one node checks deadlines and triggers notifications
notifications are queued, so they can be executed by the whole cluster
optionally supply a secret token when configuring your services, so the ping messages can't be spoofed easily

Quickstart

Up and running in less than 1 minute:

# start deadman-switch
docker run --name deadman-switch -d --rm -p 8080:8080 trusch/deadman-switch:latest

# configure service
curl -u admin:admin -XPOST --data-binary @- localhost:8080/config <<EOF
{
  "id": "service-1",
  "timeout": "30s",
  "debounce": "1m",
  "alertNotifications": [
	{
	  "type": "webhook",
	  "config": {
	    "method": "GET",
		"url": "http://localhost:8080/log?service-1-alert"
	  }
	}
  ]
}
EOF

# call the ping endpoint
curl http://localhost:8080/ping/service-1

# look at the logs
docker logs -f deadman-switch

Build and run

Dependencies

This repo requires podman and buildah as development toolset.

Ubuntu install commands:

. /etc/os-release
echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/ /" | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/Release.key | sudo apt-key add -
sudo apt-get update
sudo apt-get -y upgrade
sudo apt-get -y install podman buildah skopeo
echo "${USER}:100000:65535" | sudo tee -a /etc/subuid
echo "${USER}:100000:65535" | sudo tee -a /etc/subgid

Build image

make image

Run a local test deployment

make run

This will bring up a pod with etcd as storage backend, caddy as ingress router and two instances of deadman-switch. The pod will expose port 8080 to serve our HTTP API. You can now for example list all configured services like this:

curl -u admin:admin http://localhost:8080/config | jq .

You can also POST or DELETE service config objects using this endpoint:

curl -XPOST -u admin:admin -d '{"id":"new_service", "timeout":"10s", "notifications":[{"webhook": {"url": "https://google.com", "method": "GET"}}]}' http://localhost:8080/config
curl -XDELETE -u admin:admin http://localhost:8080/config/new_service

To actually send a ping to the deadman switch do something like this:

curl http://localhost:8080/ping/svc1?token=secret1

If you don't do anything, the application will start calling its configured webhooks after 30 seconds. You can see that in the logs: podman logs -fn deadman-switch-1 deadman-switch-2. Please note that only one of the two nodes checks the deadlines, but both nodes are used to send out the actual notification webhooks.

trusch / deadman-switch