[bitnami/etcd] Upgrading from older chart version of etcd such as 8.5.8 to the latest 10.2.11 with no downtime/data loss.

[Noy Badani]

Name and Version

bitnami/etcd 10.2.12

What is the problem this feature will solve?

In the previous etcd chart - 8.5.8, in the statefulset template, there was no mandatory matchLabels under the spec.selector.
Now, in the latest version, the statefulset template requires to have the app.kubernetes.io/component: etcd matchLabels under the spec.selector section.
Following the differences in the charts in the scenario I mentioned above - an intervention is needed to perform the upgrade, as mentioned in your documentation in the upgrading section to chart 9.x.x and above - a new label was added app.kubernetes.io/component=etcd to the StatefulSet and pods. Due to this change, the StatefulSet will be replaced (as it's impossible to add additional spec.selector.matchLabels to an existing StatefulSet) and the pods will be recreated.
The fact that matchLabels field is immutable in k8s statefulsets, and there is a need to delete the current statefulset and have orphan pods till a new statefulset is created may cause downtime in a production environment and also can potentially damage existing data during the replacement process.

Once the app.kubernetes.io/component: etcd matchLabels under the spec.selector section will not be mandatory, no manual changes will be required.

What is the feature you are proposing to solve the problem?

Removing the mandatory app.kubernetes.io/component: etcd matchLabels under the spec.selector section from the statefulset template. As was in previous versions before the 9.0.0 chart version.

What alternatives have you considered?

Performing upgrade that may include a downtime in production and also can potentially damage existing data during the replacement process - a solution we would like to avoid.

[Javier J. Salmerón García]

Hi,

I'm afraid that adding this label was part of the Bitnami standardization, and was added to a major bump because of the reasons you specify in your post. If we remove the label, wouldn't this force another major bump, causing the issues you mentioned to those upgrading from 9.x.x or 10.x.x?

[Noy Badani]

Hi @javsalgar,
Thank you for the response.
Regarding what you shared, I do understand it. However, I meant to have the option of choosing whether to use or not the "app.kubernetes.io/component: etcd" label - based on my current chart. Not to remove it entirely from the chart.
Please allow me to clarify by showing you my suggestion:
statefulset.yaml -

metadata:
  name: {{ include "common.names.fullname" . }}
  namespace: {{ .Release.Namespace | quote }}
  labels: 
    {{- include "common.labels.standard" ( dict "customLabels" .Values.commonLabels "context" $ ) | nindent 4 }}
    {{- if hasKey .Values.commonLabels "app.kubernetes.io/component" }}
    app.kubernetes.io/component: etcd
    {{- end }}
  {{- if .Values.commonAnnotations }}
  annotations: 
    {{- include "common.tplvalues.render" ( dict "value" .Values.commonAnnotations "context" $ ) | nindent 4 }}
  {{- end }}
spec:
  replicas: {{ .Values.replicaCount }}
  {{- $podLabels := include "common.tplvalues.merge" ( dict "values" ( list .Values.podLabels .Values.commonLabels ) "context" . ) }}
  selector:
    matchLabels: 
      {{- include "common.labels.matchLabels" ( dict "customLabels" $podLabels "context" $ ) | nindent 6 }}
      {{- if hasKey $podLabels "app.kubernetes.io/component" }}
      app.kubernetes.io/component: etcd
      {{- end }}
  serviceName: {{ printf "%s-headless" (include "common.names.fullname" .) | trunc 63 | trimSuffix "-" }}
  podManagementPolicy: {{ .Values.podManagementPolicy }}
  updateStrategy: 
    {{- include "common.tplvalues.render" (dict "value" .Values.updateStrategy "context" $ ) | nindent 4 }}
  template:
    metadata:
      labels: 
        {{- include "common.labels.standard" ( dict "customLabels" $podLabels "context" $ ) | nindent 8 }}
        {{- if hasKey $podLabels "app.kubernetes.io/component" }}
        app.kubernetes.io/component: etcd
        {{- end }}

This approach allows us to upgrade the chart without worrying about breaking the immutability of the StatefulSet's matchLabels.
The label "app.kubernetes.io/component: etcd" is only applied if it was already part of our existing labels, ensuring backward compatibility with our current setup.
This chart's modification should allow us to perform the upgrade smoothly without requiring manual label adjustments on our existing Kubernetes resources. And should not affect or cause issues to those upgrading from 9.x.x or 10.x.x.

It is important for me to note that I checked this on my side and it seems that this solution works as expected. First, it verifies if this label exists before adding it.

During the tests, I noticed that all the Kubernetes resources are affected by this label and the change must be contained on them as well (networkpolicy.yaml, serviceaccount.yaml, secrets.yaml, snapshot-pvc.yaml, svc-headless.yaml, svc.yaml ).

The solution proposed above can greatly help additional customers who plan to update to the latest version in a production environment without any concern or manual change on their part, while customers who have already made the update do not need to worry about another change or the removal of this label.

Please go through the proposed solution and consider adding it.
Thank you in advance.

[Juan Ariza Toledano]

Hi @noyb-jfrog

In order to adopt the approach you're suggesting, we'll have to default value for commonLabels parameter which has a series of implications. Also, please note the value for the label isn't the same one on every K8s manifest. For instance, if you check bitnami/etcd/templates/prometheusrule.yaml, you'll find there we use:

    app.kubernetes.io/component: metrics

We understand is not the best UX to ask users to perform manual steps to upgrade, that's why we bumped the major version and explicitly documented the steps to follow at https://github.com/bitnami/charts/blob/main/bitnami/etcd/README.md#to-900

[Noy Badani]

Thank you @juan131