Support Rolling Upgrades for the cockroachdb installed through helm chart
prafull01 opened this issue · comments
Rolling Upgrade for cockroach db statefulset:
Every pod of statefulset should be upgraded one by one and it should wait for the cockroach db cluster to be ready state before upgrading to the new cluster.
Currently, helm chart will upgrade the statefulset in rolling upgrade but it doesn't wait for cockroach cluster to be ready before it upgrades to the next replica.
Proposed Solution:
We already have self-signer utility for certificate management, we can expand that utility to do rolling start as well. This utility will have a sub command of rolling-restart and this will upgrade the statefulset into rolling restart fashion and wait for cockroach db cluster to be ready before upgrading the next pod. We are already doing this for the cockroach operator
This utility will be invoked as pre-upgrade command and can be enabled or disable through the values.yaml.
@udnay Please share your thoughts on this
This approach seems fine to me, @chrisseto @pseudomuto anything totally off with this before @prafull01 writes up a more detailed RFC?