dotdc / grafana-dashboards-kubernetes

A set of modern Grafana dashboards for Kubernetes.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[bug] default resolution is too low

uhthomas opened this issue · comments

commented

Describe the bug

The default resolution of 30s is too low and renders some dashboards with "No Data". This is likely because I'm using Grafana Mimir, as opposed to a standard Prometheus install.

image

How to reproduce?

  1. Collect metrics with Grafana Mimir.
  2. Load the dashboard.

Expected behavior

Changing the resolution from 30s to 1m shows the data as expected.

image

Additional context

No response

I added the resolution variable in an attempt to see spikes in a very specific case.
I think it would make more sense to remove it completely and let $__rate_interval do its job.

In case anyone needs it, it's pretty easy to edit a panel to add it back.

commented

@dotdc In general, I would agree. Though I've seen the rate internal variable also not work and I don't think it's configurable by the user?

commented

For example, these two panels behave differently without an explicit resolution. They both use auto min resolution and the rate interval, but one renders okay and the other doesn't.

image

The $__rate_interval is not configurable, but it should work properly if you remove the $resolution variable in Min step.

What is your Scrape Interval?
Do these panels work with a lower resolution set in Min step (1m)?

If it's the case, maybe the issue is tied to Grafana Mimir and/or your configuration. Maybe it's worth opening an issue there too.

I'll keep the PR as is for now, but I would rather delete the $resolution variable or set the default to 15s rather than 1m.

Doc: https://grafana.com/docs/grafana/latest/datasources/prometheus/template-variables/#use-__rate_interval

commented

Hmm, Grafana agent is configured with a 15s scrape interval.

https://github.com/uhthomas/automata/blob/ed25c6b7a525182f1c626f1fb2feda99b268db65/k8s/unwind/grafana_agent/grafana_agent_list.cue#L22

Though, changing the value to 1m in Grafana for the datasource does seem to fix things! Thanks for the pointers and the help.

Interesting...
Will add this to the "known issues" list. Feel free to share any further knowledge you may have on this issue in the future.

If anyone has further knowledge on this issue, feel free to open a PR to update the documentation.