[kube-prometheus-stack] Using Grafana's alertmanager
netikras opened this issue · comments
Is your feature request related to a problem ?
Okay, it took me a while to figure out why the hell I can't (using this chart):
- easily configure SMTP configuration in Grafana's alertmanager
- use Grafana's alertmanager to send email alerts based on alerting rules that come in the box with this chart
If my understanding is correct, all these alerting rules
are created with CRDs prometheusrules.monitoring.coreos.com. That's alright, I really don't mind having a fine set of alerting rules that come along this helm chart!
However, the problem I had was Grafana's alertmanager not picking ANY of those rules.
I mean, I have a solid heap of alerting rules and NONE of them seem to be available for alerting..
If my understanding is correct, Grafana can DISPLAY those rules, but it can't hook them up to its alertmanager; instead, I have to use Prometheus' alertmanager (configurable using prometheus CRDs, like alertmanagerconfigs.monitoring.coreos.com, alertmanagers.monitoring.coreos.com). Is that correct?
If so, is there any way to use PrometheusRules with Grafana's alertmanager?
Thing is, if I reside to Prometheus alertmanager, I'm tied to using Prometheus. However, in Grafana I have several datasources I'd like use for dashboards and alerting (e.g. Loki). And the client is not always very keen on going into the manifest modification and/or creation just to alter alerting (e.g. less technical folks, like BAs, QAs, POs, etc.). This is where Grafana's UI-based alertmanager configuration shines -- ANYONE can make a few mouseclicks, copy-paste a dashboard panel's query and hook it up to a preconfigured notification channel (Grafana appraoch). But not anyone has the skill, will or even sufficient permissions to go the kubectl way (Prometheus approach)..
If Grafana could "understand" and use PrometehusRule CRDs for alerting, it would be the best of both worlds: rules could be created via CRDs automatically and/or via Grafana manually and they all would be routed through the same Grafana's alertmanager.
Describe the solution you'd like.
A way to use PrometheusRule CRDs with Grafana's alertmanager
Describe alternatives you've considered.
I've considered using Prometheus alertmanager, of course, but AFAIK it's only configurable by technical folks (through CRDs). And the rules can't be disabled/modified via Grafana (e.g. kube-proxy rule, it's firing all the time -- well of course it is! Cilium CNI does not use kube-proxy. I'd like to mute it, but that means diving into kubectl, meaning it is not always doable by the client itself).
Also, I (/the client) can't see in Grafana which alerts will be routed to email (critical alerts), which will hit Teams (business alerts), and which ones will land on Slack (technical alerts).
Additional context.
This helm chart comes with a solid set of alerting rules. It would be a shame if I had to rewrite them all into Grafana just to be able to use them with Grafana's alertmanager (although it might be the only way...?).
It's perfectly possible the solution already exists, I just couldn't find it. If you could point me in the right direction, I'd greatly appreciate it and happily go on my way w/o bothering you any more!
A way to use PrometheusRule CRDs with Grafana's alertmanager
Some something that you have raise at https://github.com/grafana/grafana-operator
We only maintain a chart here, not operators.
The problem with this request is that the configuration for grafana alerting rules is totally different from the alertmanager/prometheusrules.
Even grafana themselves do not really support it, cortextool and alloy both import the rules to mimir not to grafana.