Grafana's /alerting/list doesn't show any alerts via datasource plugin
chouzee opened this issue · comments
Hi! I try to list alerts in Grafana using the plugin as a datasource but nothing shows up but they are show when I don't use the plugin. The url for both datasources are the same. Could you clarify if the plugin supports this feature?
Forgot to mention: for both datasource I turned on Manage alerts via Alerting UI
Hi @chouzee ! I have checked your question.
- I created a test alert
- Went to alerting tab
- I Checked the
Listtab, and I had no error here
@chouzee can you share what version you use? Your error said that something was wrong with the network.
@dmitryk-dk
this one feature VictoriaMetrics/VictoriaMetrics#1739 and VictoriaMetrics/VictoriaMetrics#2583
like, read rules, alerts from VMalert. It worked in datasource prometheus
@yuriydzobak ! Thank you, I will look at those issues and test them.
@dmitryk-dk I can confirm alerting rules aren't working with VictoriaMetrics DS plugin. It needs to be fixed. It is likely, plugin doesn't forward alerts request to the VictoriaMetrics backend.
Steps to reproduce:
- Setup Grafana with VictoriaMetrics plugin
- Spin up VictoriaMetrics single node
./bin/victoria-metrics --vmalert.proxyURL=http://localhost:8880 - Spin up vmalert with some alerting rules
./bin/vmalert -rule=/alerting/rules -datasource.url=http://localhost:8428 -notifier.url=http://localhost:9093 - Open Grafana and go to Alerting tab
Expected result:
Actual result:
Please note, both datasources are pointing to the same URL
Hi @hagen1778 , @yuriydzobak , @chouzee ! I have found the difference between grafana plugin and our implementation. I think we have a possible solution and @Loori-R will try to do it on FE part
Hi @hagen1778 , @yuriydzobak , @chouzee ! I have investigated the issue and created two issues in the Grafana repository. The main problem is when data source type differs from prometheus or loki grafana only calls internal API for alerts created from the Grafana UI.
There are two issues with details explanations.
grafana/grafana#67466
grafana/grafana#68169
So it is like Grafana doesn't make the API call to fetch the alerts if your datasource isn't prometheus or loki? Is my understanding correct?
So it is like Grafana doesn't make the API call to fetch the alerts if your datasource isn't
prometheusorloki? Is my understanding correct?
They make API call, but they use the default source name grafana when to build URL to fetch
`/api/prometheus/${getDatasourceAPIUid(dataSourceName)}/api/v1/rules`,
function getDatasourceAPIUid(dataSourceName) returns grafana instead of the datasource_ID.
But even if we do some tricks with Grafana code the backend code has some conditions which prevent the request to data source different from Prometheus or loki.
When I remove those conditions and add datasource type like victoriametrics to the places where grafana checks datasource type it works as expected.
That seems like a fix for the issue VictoriaMetrics/VictoriaMetrics#1739 (comment)
-vmalert.proxyURL string
Optional URL for proxying alerting API requests from Grafana. For example, if -vmalert.proxyURL is set to http://vmalert:8880 , then requests to /api/v1/rules are proxied to http://vmalert:8880/api/v1/rules
vmalert.proxyURL
Hi @vainkop ! I don't think this will help, because Grafana itself determines where to send the request. The problem is that Grafana stores encoded data that supports aletings
vmalert.proxyURL
Hi @vainkop ! I don't think this will help, because Grafana itself determines where to send the request. The problem is that Grafana stores encoded data that supports aletings
So at the moment the Grafana (6.59.*) used in https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-k8s-stack can be considered as not full compatible with VMalert (v1.94.0) from the same chart? :(