[prometheus-windows-exporter] Readiness probe always returning 404
petewall opened this issue · comments
Describe the bug a clear and concise description of what the bug is.
I am deploying Windows exporter onto many clusters with windows nodes, both AKS and EKS. In both circumstances, the readiness probe is returning 404 when trying to access /
, which means the pod never becomes ready and it goes into CrashLoopBackOff:
The pod events:
Events:
Type Reason Age From Message
---- ------ ---- ---- ------- Normal Scheduled 17m default-scheduler Successfully assigned default/grafana-k8s-monitoring-prometheus-windows-exporter-wwrlb to ip-192-168-62-144.us-west-2.compute.internal
Normal Pulled 17m kubelet Container image "ghcr.io/prometheus-community/windows-exporter:0.26.1" already present on machine
Normal Created 17m kubelet Created container configure-firewall
Normal Started 17m kubelet Started container configure-firewall
Normal Pulled 16m (x2 over 17m) kubelet Container image "ghcr.io/prometheus-community/windows-exporter:0.26.1" already present on machine
Normal Killing 16m kubelet Container windows-exporter failed liveness probe, will be restarted
Normal Created 16m (x2 over 17m) kubelet Created container windows-exporter
Normal Started 16m (x2 over 17m) kubelet Started container windows-exporter
Warning Unhealthy 16m (x4 over 16m) kubelet Liveness probe failed: HTTP probe failed with statuscode: 404
Warning BackOff 7m12s (x32 over 14m) kubelet Back-off restarting failed container
Warning Unhealthy 2m4s (x63 over 17m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 404
I believe this is because Windows Exporter itself does not respond to /
in its webserver:
https://github.com/prometheus-community/windows_exporter/blob/master/exporter.go#L231-L274
What's your helm version?
version.BuildInfo{Version:"v3.15.3", GitCommit:"3bb50bbbdd9c946ba9989fbe4fb4104766302a64", GitTreeState:"clean", GoVersion:"go1.22.5"}
What's your kubectl version?
Client Version: v1.30.3 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.30.0
Which chart?
prometheus-windows-exporter
What's the chart version?
0.4.0
What happened?
Readiness probe fails with 404
What you expected to happen?
Readiness probe should use a path that exists in the http webserver
How to reproduce it?
Deploy promethus-windows-exporter
Enter the changed values of values.yaml?
The only changes from the default:
config: |-
collectors:
enabled: cpu,cs,container,logical_disk,memory,net,os
collector:
service:
services-where: "Name='containerd' or Name='kubelet'"
Enter the command that you execute and failing/misfunctioning.
helm install is succeeding, the pod fails to stay online
Anything else we need to know?
Before Kubernetes stops the pod, it does respond to requests to /metrics
, which means the http service is running.
Looks like this probably was the reason that /
doesn't work anymore:
prometheus-community/windows_exporter@b977c84