Rerunning the cockroachdb-self-signer job breaks the cluster
jamiechapmanbrn opened this issue · comments
If somehow the cockroachdb self signer job is ever deleted, on reapply the new job will overwrite the existing secret, meaning that any time cockroachdb is rolled after that the new node will fail to come up without a manuall restart of the cluster.
This also causes lack of trust in any user generated certificates, requiring they be rotated as well.
It would be much nicer if the container in the job checked if there was a known good certificate already, and printed an obvious error message that there already is a certificate, and if you want to regenerate delete the secret first.
Did some more digging, turns out the problem was the self-signer-cleanup job. It was a post-delete hook, and wasn't supposed to be run at all. Helm template happily generated it and I didn't realize it wasn't supposed to run at all.
I'm closing the issue, since it's clear that the bug isn't really here, but more an issue in helm template.