[bitnami/kube-prometheus] UPGRADE FAILED: create: failed to create: Secret "sh.helm.release.v1.kube-prometheus.v3" is invalid: data: Too long: must have at most 1048576 bytes

[Raúl Macián]

Name and Version

bitnami/kube-prometheus:9.5.12

What architecture are you using?

amd64

What steps will reproduce the bug?

1. Create a few prometheus scrapping config and prometheus rules and add them to values.yaml:
2. Launch helm with those configs

Are you using any custom parameters or values?

yes, feeding the prometheus rules with a jinja template

    additionalScrapeConfigs:
      enabled: true
      type: internal
      internal:
        jobList: "{{ prometheus_scrape_config }}"
    additionalPrometheusRules: "{{ prometheus_rules }}"

What is the expected behavior?

Installation successfull

What do you see instead?

Release "kube-prometheus" does not exist. Installing it now.
Error: create: failed to create: Secret "sh.helm.release.v1.kube-prometheus.v1" is invalid: data: Too long: must have at most 1048576 bytes

Additional information

Since kubernetes has a limit of 1Mb for a secret and is mandatory that scrapping configs and rules are stored in a Secret is almost impossible to have a production configuration.
The problem is not a high volume of rules or scrapping configs. I can render my values.yaml with helm template and it's only about 112kb and I can apply it with kubectl. The problem I think is the chart size itself. Helm sends the whole chart as a secret to mantain history and it's very easy to reach this point.

Why cannot be stored in a configmap to avoid this limitation?

[Javier J. Salmerón García]

Hi,

Thank you for reporting! This is something that has appeared in our testing too and we're trying to find a solution. About storing it in a configmap, that's something to be answered by the helm developers, as this is as limitation on their side. I imagine that, by default, they may to store it in a secret because there may be passwords implied.