Secret not yet active when deploying with Helm
Clemens85 opened this issue · comments
Hi there,
firstly I want to mention that my issue is more a discussion / question than a bug report (I think so at least):
We deploy all our applications with Helm and we manage our secrets inside our repository with sealed-secrets. Therefore we create them lcoally with the kubeseal command in such a way that they are part of the Helm chart of an application that we want to deploy later on.
Our applications get the secret values then injected as env variables, which are part of the deployment (and denoted in the Helm deployment.yaml template).
When we deploy such an application with Helm, the order of resource creation is correct, so it firstly deploys the (sealed-)secrets and then the deployment Pods. This worked quite good for ~1 year.
Now we noticed however, that there is an issue: When we want to deploy e.g. an updated secret (also with a new application version, which means that our Pods are also newly deployed), some new created Pods get the old secret value and some Pods get the new secret value.
I think that we also saw the issue: It seems like that the Secret is not yet in "Active" state when our first (new) Pods are starting up, which means that they get the old value, and at some point of time the (updated) Secret is ready, so the Pods that are started after this point of time get also the correct new Secret value.
At least that's what we saw, but this might also be a wrong analysis. So my first question: Does that sound plausible? And/or are you aware of this issue? I googled for this issue, but didn't found anything to suggest that people faced a similar issue... so I am not quite sure if we are just doing something completly wrong....
I am not a Helm expert, but after some recherches it seems to me that Helm does indeed not wait till the resources are really ready in the deploy process. So I am wondering what we could do to solve this issue properly? (Of course we could just deploy twice, but this is more a workraound for me). At a glance, I see two options:
- Deploy the secrets independently before the actual application deployment (like an own deploy artefact), which ensures that they are really active, before we proceed to the actual application depoyment.
- I read something about initContainers which can be used to wait for other resources being really ready, but I am not sure if this really helps here, because the secrets might already exist, but not with the propery new value during deployment.
Best regards
Clemens
This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.
Due to the lack of activity in the last 7 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.