prometheus-community / helm-charts

Prometheus community Helm charts

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[prometheus-windows-exporter] Readiness probe always returning 404

petewall opened this issue · comments

Describe the bug a clear and concise description of what the bug is.

I am deploying Windows exporter onto many clusters with windows nodes, both AKS and EKS. In both circumstances, the readiness probe is returning 404 when trying to access /, which means the pod never becomes ready and it goes into CrashLoopBackOff:

The pod events:

Events:
Type     Reason     Age                   From               Message
----     ------     ----                  ----               -------                                                                                                                                     Normal   Scheduled  17m                   default-scheduler  Successfully assigned default/grafana-k8s-monitoring-prometheus-windows-exporter-wwrlb to ip-192-168-62-144.us-west-2.compute.internal
Normal   Pulled     17m                   kubelet            Container image "ghcr.io/prometheus-community/windows-exporter:0.26.1" already present on machine
Normal   Created    17m                   kubelet            Created container configure-firewall
Normal   Started    17m                   kubelet            Started container configure-firewall
Normal   Pulled     16m (x2 over 17m)     kubelet            Container image "ghcr.io/prometheus-community/windows-exporter:0.26.1" already present on machine
Normal   Killing    16m                   kubelet            Container windows-exporter failed liveness probe, will be restarted
Normal   Created    16m (x2 over 17m)     kubelet            Created container windows-exporter
Normal   Started    16m (x2 over 17m)     kubelet            Started container windows-exporter
Warning  Unhealthy  16m (x4 over 16m)     kubelet            Liveness probe failed: HTTP probe failed with statuscode: 404
Warning  BackOff    7m12s (x32 over 14m)  kubelet            Back-off restarting failed container
Warning  Unhealthy  2m4s (x63 over 17m)   kubelet            Readiness probe failed: HTTP probe failed with statuscode: 404

I believe this is because Windows Exporter itself does not respond to / in its webserver:
https://github.com/prometheus-community/windows_exporter/blob/master/exporter.go#L231-L274

What's your helm version?

version.BuildInfo{Version:"v3.15.3", GitCommit:"3bb50bbbdd9c946ba9989fbe4fb4104766302a64", GitTreeState:"clean", GoVersion:"go1.22.5"}

What's your kubectl version?

Client Version: v1.30.3 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.30.0

Which chart?

prometheus-windows-exporter

What's the chart version?

0.4.0

What happened?

Readiness probe fails with 404

What you expected to happen?

Readiness probe should use a path that exists in the http webserver

How to reproduce it?

Deploy promethus-windows-exporter

Enter the changed values of values.yaml?

The only changes from the default:

config: |-
  collectors:
    enabled: cpu,cs,container,logical_disk,memory,net,os
  collector:
    service:
      services-where: "Name='containerd' or Name='kubelet'"

Enter the command that you execute and failing/misfunctioning.

helm install is succeeding, the pod fails to stay online

Anything else we need to know?

Before Kubernetes stops the pod, it does respond to requests to /metrics, which means the http service is running.

Looks like this probably was the reason that / doesn't work anymore:
prometheus-community/windows_exporter@b977c84