[kube-prometheus-stack] Grafana "Use an existing secret for the admin user" used to change password?
dcfitzpatrick00 opened this issue · comments
Describe the bug a clear and concise description of what the bug is.
My values.yaml has the following to use an existingSecret for Grafana:
grafana:
admin:
existingSecret: "kube-prometheus-stack-grafana"
userKey: admin-user
passwordKey: admin-password
I'm using SealedSecrets / AgroCD - To deploy an UPDATED SealedSecret file with new Grafana password. The deployment works but Grafana ignores the new password within the secret. Grafana is still accepting the original password from the previous version of the secret, from the initial install. Trying to determine if this is by design or bug.
What's your helm version?
$ helm version version.BuildInfo{Version:"v3.10.2", GitCommit:"50f003e5ee8704ec937a756c646870227d7c8b58", GitTreeState:"clean", GoVersion:"go1.18.8"}
What's your kubectl version?
$ kubectl version WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3+k3s1", GitCommit:"f2585c1671b31b4b34bddbb3bf4e7d69662b0821", GitTreeState:"clean", BuildDate:"2022-10-25T19:59:38Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.7 Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3+k3s1", GitCommit:"f2585c1671b31b4b34bddbb3bf4e7d69662b0821", GitTreeState:"clean", BuildDate:"2022-10-25T19:59:38Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/amd64"}
Which chart?
kube-prometheus-stack
What's the chart version?
41.7.4
What happened?
Troubleshooting:
This is the only secret I have with the word grafana in it:
$ k get secrets -A | grep grafana
monitoring kube-prometheus-stack-grafana Opaque 3 2d1h
The admin ID & password are defined within the secret:
$ k describe secrets -n monitoring kube-prometheus-stack-grafana
Name: kube-prometheus-stack-grafana
Namespace: monitoring
Labels:
Annotations:
Type: Opaque
Data
admin-password: 17 bytes
admin-user: 5 bytes
ldap-toml: 0 bytes
I can even inspect the secret and see SealedSecrets has applied the new password (I've changed it again since):
$ kubectl get secret --namespace monitoring kube-prometheus-stack-grafana -o jsonpath='{.data.admin-password}' | base64 -d
ZvtZwNRGsFU5nsLOz
I only have a single Grafana pod, I've restart it to force it to pick up the new password:
$ k get pods -A | grep grafana
monitoring grafana-5c84cc4c77-jtdqp 3/3 Running 0 51s
When I describe the pod, it states the ENV variable will be set from the secret:
Environment:
GF_SECURITY_ADMIN_USER: <set to the key 'admin-user' in secret 'kube-prometheus-stack-grafana'> Optional: false
GF_SECURITY_ADMIN_PASSWORD: <set to the key 'admin-password' in secret 'kube-prometheus-stack-grafana'> Optional: false
I can view the ENV variables passed inside the Pod showing the new password is being picked up:
$ k exec -n monitoring grafana-5c84cc4c77-jtdqp -c grafana -- env | grep ADMIN
GF_SECURITY_ADMIN_USER=admin
GF_SECURITY_ADMIN_PASSWORD=ZvtZwNRGsFU5nsLOz
However, the Grafana dashboard ignored new password; only the original password "prom-operator" is being accepted for account "admin".
Where is it getting this old password from? I assume it stored in the database and the secrets are initial install only and then no longer used? I was unable to find this information in the documentation.
What you expected to happen?
I expected the admin password to be updated based on the value stored within the secret.
How to reproduce it?
Change password stored within Grafana secret and restart the pod. The ENV variable GF_SECURITY_ADMIN_PASSWORD reflects the new password but is ignored by Grafana.
Enter the changed values of values.yaml?
No response
Enter the command that you execute and failing/misfunctioning.
Grafana Dashboard only accepts initial secret password value, not updated password.
Anything else we need to know?
I could understand if this is by design, but I can't find a reference saying that Kubernetes secret is used for Grafana initial install only and ignored afterwards.
Kube prom stack uses the upstream grafana chart. You should ask there: https://github.com/grafana/helm-charts/tree/main/charts/grafana