How to have $external_labels show up in discord?
ki4rbc opened this issue · comments
I am attempting to use an external label, locID, with Prometheus alerts. I have the external label being sent to alertmanager-discord, but how does one configure what's going to discord; other then modifying main.go?
Here is the print-pretty formatted input JSON:
{
"receiver": "prometheus-msteams",
"status": "firing",
"alerts": [{
"status": "firing",
"labels": {
"alertname": "A test alert",
"job": "kubernetes-apiservers",
"locID": "00000000-0000-0000-0000-000000000000",
"severity": "notice"
},
"annotations": {
"description": "There are 1 kubernetes-apiservers jobs running",
"summary": "This is only a test on locationID: "
},
"startsAt": "2020-08-31T18:36:25.881534544Z",
"endsAt": "0001-01-01T00:00:00Z",
"generatorURL": "http://prometheus-subscriber-server-6ccf748db-s4j9s:9090/graph?g0.expr=sum+by%28job%29+%28up%29+%3E+0\u0026g0.tab=1"
}, {
"status": "firing",
"labels": {
"alertname": "A test alert",
"job": "kubernetes-nodes",
"locID": "00000000-0000-0000-0000-000000000000",
"severity": "notice"
},
"annotations": {
"description": "There are 2 kubernetes-nodes jobs running",
"summary": "This is only a test on locationID: "
},
"startsAt": "2020-08-31T18:35:25.881534544Z",
"endsAt": "0001-01-01T00:00:00Z",
"generatorURL": "http://prometheus-subscriber-server-6ccf748db-s4j9s:9090/graph?g0.expr=sum+by%28job%29+%28up%29+%3E+0\u0026g0.tab=1"
}, {
"status": "firing",
"labels": {
"alertname": "A test alert",
"job": "kubernetes-nodes-cadvisor",
"locID": "00000000-0000-0000-0000-000000000000",
"severity": "notice"
},
"annotations": {
"description": "There are 2 kubernetes-nodes-cadvisor jobs running",
"summary": "This is only a test on locationID: "
},
"startsAt": "2020-08-31T18:35:25.881534544Z",
"endsAt": "0001-01-01T00:00:00Z",
"generatorURL": "http://prometheus-subscriber-server-6ccf748db-s4j9s:9090/graph?g0.expr=sum+by%28job%29+%28up%29+%3E+0\u0026g0.tab=1"
}, {
"status": "firing",
"labels": {
"alertname": "A test alert",
"job": "kubernetes-service-endpoints",
"locID": "00000000-0000-0000-0000-000000000000",
"severity": "notice"
},
"annotations": {
"description": "There are 1 kubernetes-service-endpoints jobs running",
"summary": "This is only a test on locationID: "
},
"startsAt": "2020-08-31T18:35:25.881534544Z",
"endsAt": "0001-01-01T00:00:00Z",
"generatorURL": "http://prometheus-subscriber-server-6ccf748db-s4j9s:9090/graph?g0.expr=sum+by%28job%29+%28up%29+%3E+0\u0026g0.tab=1"
}, {
"status": "firing",
"labels": {
"alertname": "A test alert",
"job": "prometheus",
"locID": "00000000-0000-0000-0000-000000000000",
"severity": "notice"
},
"annotations": {
"description": "There are 1 prometheus jobs running",
"summary": "This is only a test on locationID: "
},
"startsAt": "2020-08-31T18:36:25.881534544Z",
"endsAt": "0001-01-01T00:00:00Z",
"generatorURL": "http://prometheus-subscriber-server-6ccf748db-s4j9s:9090/graph?g0.expr=sum+by%28job%29+%28up%29+%3E+0\u0026g0.tab=1"
}],
"groupLabels": {
"alertname": "A test alert"
},
"commonLabels": {
"alertname": "A test alert",
"locID": "00000000-0000-0000-0000-000000000000",
"severity": "notice"
},
"commonAnnotations": {
"summary": "This is only a test on locationID: "
},
"externalURL": "",
"version": "4",
"groupKey": "{}:{alertname=\"A test alert\"}"
}```
Cheers for the input json context,
Right now there is not a way for you to do this, The way I do this in my infra is to inline the interesting data bits into my alerts description
What would you want to do with the locID? Just display it?
Thanks for responding so quickly,
The locID is how I know where the alert is coming from it comes from an os env which is then used as a yaml anchor which is later used to create the external_label.
The problem I am running into is that external_labels are not available for processing within alert rules; unless I am missing something.
And yes. I would like locID to show up anywhere in the alert in discord.
Hello again @benjojo,
So I got this mod of main.go to work but I'm not entirely happy and it's not something I think I would suggest a fork for.
--- original/alertmanager-discord/main.go 2020-08-31 18:35:49.698522600 -0400
+++ alertmanager-discord/main.go 2020-08-31 22:28:46.702549100 -0400
@@ -121,7 +121,7 @@
}
RichEmbed.Fields = append(RichEmbed.Fields, discordEmbedField{
- Name: fmt.Sprintf("[%s]: %s on %s", strings.ToUpper(status), alert.Labels["alertname"], realname),
+ Name: fmt.Sprintf("[%s]: %s on %s - %s", strings.ToUpper(status), alert.Labels["alertname"], realname, alert.Labels["locID"]),
Value: alert.Annotations.Description,
})
}
I found a work around to adding an alert origin identity, locationID in my case. By creating the alertmanager alerting rules is a config map template, I have access to environmental variables. It took a while to figure out I have to escape curly brackets. For example
- alert: Deployments unavailable
expr: 100 * (kube_deployment_status_replicas_available / kube_deployment_status_replicas) != 100
for: 5m
labels:
severity: warn
annotations:
summary: 'Deployments unavailable at Location ID {{ .Values.locationID }}'
description: '{{ "{{" }} $labels.deployment {{ "}}" }}.{{ "{{" }} $labels.namespace {{ "}}" }} at {{ "{{" }} $value | printf "%.2f" {{ "}}" }}% of intent'I now can see where the alerts originate from.
I did see that on some alerts there is no label instance or exported_instance so we can still end up with alerts where realname is NULL.
I've created a fork and will work though a solution to propose.