secret too long with helm-operator but manual helm work
knack-roque opened this issue · comments
helm-operator v1.3.0
helm3 v3.5.3
I'm attempting to upgrade an existing release and the helm operator is reporting:
{"action":"upgrade","caller":"release.o:357","component":"release","error""upgrade failed: create: failed to create: Secret \"sh.helm.release.v1.<release name>.v82\" is invalid: data: Too long: must have at most 1048576 bytes","helmVersion":"v3","release":"gob","resource":"<namespace>:helmrelease/<release name>","targetNamespace":"<target namespace>","ts":"2021-11-17T13:47:54.872979451Z"}
The weird bit is if I take the values from the helm release and put them in an overrides.yaml; and then manually do a helm3 upgrade --install ... with them it works just fine. With the values and the chart we're using the manifest is large but I'm trying to figure out what the helm-operator might be adding in that's bumping it over the limit or is there some other check that the helm-operator is using that's not being done by a manual helm command.
Is there a way to get helm-operator to at least report the size of the secret it's attempting to create so I can try messing with the values to get it to reduce?
I can help you figure out how to compile Helm Operator if you want, I don't think there are any hidden debug features that would spit out extra information about the release before applying it, but I haven't checked the source to be sure.
Before I undertake to engage this further, I am duty bound to warn you that Helm Operator is effectively EOL and the migration timetable says that no new features can be added: https://fluxcd.io/docs/migration/timetable/ we are in the "security fixes and migration assistance only" stage of supporting Helm Operator users in the legacy side of Flux.
I wonder if there's a reason (other than inertia, which seems to be the most common answer when I ask) that you wouldn't upgrade to Flux v2 and Helm Controller? This project (Helm Operator) has not much life left in it, and it's unlikely that any issues you may find can be fixed, but it is very likely that those same issues have already been addressed in Helm Controller as that project and Flux v2 are both actively developed every day.
If there are any blockers that have prevented you from upgrading, I'm interested in hearing more about those! I'm not sure what extra data Helm Operator might be including in the secret, and it seems unlikely to me too, but I'm not arguing with that which you've seen with your own eyes. Sorry for the trouble around this. I would have expected the output of helm template with a complete values file matching whatever values Helm Operator receives through the HelmRelease CR, to match closely with what Helm Operator actually applies to the cluster at the end of the Render stage.
Yes inertia plus going back and retooling a lot of our deployment processes to not use it.
Sleep is a miracle drug that I should indulge more of.
I forgot that if you do a simple kubectl apply the in cluster helmrelease resource wasn't dropping the values I had removed the helmrelease.yaml. This meant I had old cruft in the cluster that was bloating the secret's data.release.