medic / cht-watchdog

Configuration for deploying a monitoring/alerting stack for CHT

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Clean up default format of alerts (if possible ;)

mrjones-plip opened this issue · comments

Now that we've enabled alerts (#14), it turns out they're very verbose.

This is what is seen right now:

[FIRING:28] DB Fragmentation CHT (cht)
FiringValue: B=27.055714740528494, C=1
Labels:
- alertname = DB Fragmentation
- db = _users
- grafana_folder = CHT
- instance = INSTANCE-HERE.medicmobile.org
- job = cht
Annotations:
- description = The [_users] database for the CHT Server [INSTANCE-HERE.medicmobile.org] is highly fragmented.
Source: https://allies-monitoring-alerting.dev.medicmobile.org/alerting/grafana/ot6lYCYVz/view?orgId=1
Silence: https://allies-monitoring-alerting.dev.medicmobile.org/alerting/silence/new?alertmanager=grafana&matcher=alertname%3DDB+Fragmentation&matcher=db%3D_users&matcher=grafana_folder%3DCHT&matcher=instance%3DINSTANCE-HERE.medicmobile.org&matcher=job%3Dcht
Dashboard: https://allies-monitoring-alerting.dev.medicmobile.org/d/oa2OfL-Vk?orgId=1
Panel: https://allies-monitoring-alerting.dev.medicmobile.org/d/oa2OfL-Vk?orgId=1&viewPanel=13

And it'd be great if we greatly condense it and remove unused info:

  • make all Long URLs markdown/HTML links for Source, Silence, Dashboard and Panel
  • remove db - _users
  • remove grafana_folder = CHT
  • remove instance = INSTANCE-HERE.medicmobile.org
  • remove empty Annotations
  • remove title for description =

Like so:

[FIRING:28] DB Fragmentation: [_users] database on INSTANCE-HERE.dev.medicmobile.org.

Value: 27.055714740528494, Alert threshold: 1

Links: Source - Silence - Dashboard - Panel

This post might help.

Notifications reduced to single-line messages:
image

TODO: Default message formatting needs to be upstreamed