sepulworld / deadman-check

Monitoring companion for Nomad periodic jobs and Cron

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Add a k/v tree mechanism to switch_monitor

sepulworld opened this issue · comments

per discussion with @hoadley -

would be great to reduce the number of switch_monitors that need to run by allowing the switch_monitor to traverse a key/value tree. Along with the tree traversal, we need to add more information to the key being set. We need to set a JSON object in the key that looks like this

{
	"epoch": 1492482679470,
	"frequency": 3000
}

Let's say we have have 3 distinct periodic services "SilverBullet", "GreyBullet", "GoldBullet"

Here are the target keys to be updated and monitored
deadman/DeadmanSetSilverBulletPeriodicProcess
deadman/DeadmanSetGreyBulletPeriodicProcess
deadman/DeadmanSetGoldBulletPeriodicProcess

Each of these could contain their own unique "frequency" settings.

We could spawn one switch_monitor to monitor the deadman/ k/v tree like so:

job "DeadmanMonitoring" {
  datacenters = ["dc1"]
  type = "service"

  group "monitor" {
    task "DeadmanMonitorSilverBulletPeriodicProcess" {
      driver = "docker"
      config {
        image    = "sepulworld/deadman-check"
        command  = "switch_monitor"
        args     = [
          "--host",
          "10.0.0.1",
          "--port",
          "8500",
          "--key-tree",
          "deadman/",
          "--alert-to",
          "slackroom",
          "--daemon",
          "--daemon-sleep",
          "900"]
      }
      resources {
        cpu = 100
        memory = 256
      }
      env {
        SLACK_API_TOKEN = "YourSlackApiToken"
      }
    }
  }
}

This may be backwards incompatible, but perhaps we can make the changes to work along side current behavior.