K8S issue

Question

K8S issue

jr0dd opened this issue 3 years ago · comments

First off, thank you for your work on this. I'm just having issues with the dashboard when running this on my k8s cluster. More than likely a relabeling issue and I am not quite sure how to address it to be completely honest. My router is on IP 10.10.0.1 but the container IP is 172.16.0.66. I only have one relabeling, but the dashboard is showing a lot of duplicate entries. I'm definitely a noob at grafana/prometheus so hoping you could maybe point me in the right direction.

relabelings:

- sourceLabels: [__param_target]
    targetLabel: instance

screenshot:

IgorKha · Answer 1 · Wed Jul 21 2021 03:59:39 GMT+0800 (China Standard Time)

Hey! I don't think that is a problem of the dashboard, but it might relate to the configuration of the prometheus for k8s.
Here you may try to find a key to the issue: prometheus-operator/prometheus-operator#1928
Unfortunately, I am not really familiar with k8s, so this is all i could suggest

j_r0dd · Answer 2 · Wed Jul 21 2021 06:41:59 GMT+0800 (China Standard Time)

yeah definitely an issue with the labelings. i'll play around with it more later

j_r0dd · Answer 3 · Thu Jul 22 2021 01:04:45 GMT+0800 (China Standard Time)

So what happened was it was just the old metrics data overlapping. I kept reinstalling the pod when I was testing my config. I did need to use the transform of Series to Row with a value of Max on a couple of the panels because my router is a dual core and would show duplicate entries and I added CPU {{hrDeviceIndex}} for the legend on the CPU Load panel to list CPU numbers.