[kube-prometheus-stack] Prometheus is sending alerts to the alertmanager deployed in separate namespace
wrahmann opened this issue · comments
Hi,
As per prometheus helm chart documentation , the default configuration ensures that the Prometheus will send alerts to only those alertmanger which are deployed as a part of that release.
However, in my case , when we deployed two different kube-prometheus-stack , the prometheus sent alerts to other alertmanager as well.
My question is :
- If we must define alertingEndpoints to achieve ?
- Do I need to change the ports of alertmanager for both instances?
The deployment was done on Kubernetes cluster under different namespaces.
What's your helm version?
version.BuildInfo{Version:"v3.11.3", GitCommit:"66a969e7cc08af2377d055f4e6283c33ee84be33", GitTreeState:"clean", GoVersion:"go1.19.3"}
What's your kubectl version?
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", GitCommit:"9352c723d073c7b1b622f84a25959e5f5d610ad2", GitTreeState:"clean", BuildDate:"2023-06-21T10:29:31Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v5.0.1 Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", GitCommit:"9352c723d073c7b1b622f84a25959e5f5d610ad2", GitTreeState:"clean", BuildDate:"2023-06-21T10:18:33Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}
Which chart?
prometheus-kube-stack
What's the chart version?
45.9
What happened?
Two different sets of prometheus/alertmanagers were deployed under different namespaces.
What you expected to happen?
It was expected that each prometheus will send alerts to its own alertmanager cluster . However, we saw alerts from one prometheus cluster reported by the alertmaager of second cluster.
How to reproduce it?
Deploy two instance of kube prometheus . Alerts reported by one prometheus will be send to another alertmanager
Enter the changed values of values.yaml?
NONE
Enter the command that you execute and failing/misfunctioning.
helm upgrade --install prometheus -f pxy9-eccd-chf-mgmt-0001-values_promStack_45.9.yaml kube-prometheus-stack-45.9.0/kube-prometheus-stack -n pxy9-mgmt-cluster
Anything else we need to know?
None