[BUG] Rancher Monitoring Chart PrometheusRules Fail to Load
atifhafeez opened this issue · comments
Rancher Server Setup
- Rancher version: 2.8.3
- Installation option (Docker install/Helm Chart): Helm chart rancher-backup:103.0.1+up4.0.1
- If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc): RKE2
Information about the Cluster
- Kubernetes version: v1.26.10+rke2r2
- Cluster Type (Local/Downstream): Downstream
- If downstream, what type of cluster? (Custom/Imported or specify provider for Hosted/Infrastructure Provider): EC2
Describe the bug
I was using rancher monitoring helm chart "charts/rancher-monitoring/102.0.3+up40.1.2". All prometheus rules were loaded successfully. However, when i upgrade helm chart to anything higher than this chart (i.e. < charts/rancher-monitoring/103.0.0+up45.31.1), some of the prometheus rules fails to load (e.g. charts/rancher-monitoring/103.0.0+up45.31.1/templates/prometheus/rules-1.14/kube-apiserver-availability.rules.yaml and 2 more). I am calling this chart as dependency directly from public repo which means i cant do the fix in the rule.
To Reproduce
Simply install this helm chart on k8s version given above and check for prometheus rules on prometheus.
Result
Expected Result
All rules should be loaded successfully as it was happenning in previous helm charts.
Screenshots
Additional context
I'm also seeing this issue on RKE2, getting pretty constant alerts firing that Prometheus can't evaluate rules properly