[bitnami/kube-prometheus] UPGRADE FAILED: create: failed to create: Secret "sh.helm.release.v1.kube-prometheus.v3" is invalid: data: Too long: must have at most 1048576 bytes
rmacian opened this issue · comments
Name and Version
bitnami/kube-prometheus:9.5.12
What architecture are you using?
amd64
What steps will reproduce the bug?
- Create a few prometheus scrapping config and prometheus rules and add them to values.yaml:
- Launch helm with those configs
Are you using any custom parameters or values?
yes, feeding the prometheus rules with a jinja template
additionalScrapeConfigs:
enabled: true
type: internal
internal:
jobList: "{{ prometheus_scrape_config }}"
additionalPrometheusRules: "{{ prometheus_rules }}"
What is the expected behavior?
Installation successfull
What do you see instead?
Release "kube-prometheus" does not exist. Installing it now.
Error: create: failed to create: Secret "sh.helm.release.v1.kube-prometheus.v1" is invalid: data: Too long: must have at most 1048576 bytes
Additional information
Since kubernetes has a limit of 1Mb for a secret and is mandatory that scrapping configs and rules are stored in a Secret is almost impossible to have a production configuration.
The problem is not a high volume of rules or scrapping configs. I can render my values.yaml with helm template and it's only about 112kb and I can apply it with kubectl. The problem I think is the chart size itself. Helm sends the whole chart as a secret to mantain history and it's very easy to reach this point.
Why cannot be stored in a configmap to avoid this limitation?
Hi,
Thank you for reporting! This is something that has appeared in our testing too and we're trying to find a solution. About storing it in a configmap, that's something to be answered by the helm developers, as this is as limitation on their side. I imagine that, by default, they may to store it in a secret because there may be passwords implied.