Impossible to upgrade to 0.5.0

[Sébastien GLON]

Describe the bug
We use flux to deploy our workload.
We have upgraded from 0.2.0.
We have deployed new CRD.
Now we can't update our work VSO report an error on scheme validation:

manager 2024-02-16T15:05:10Z    ERROR    Reconciler error    {"controller": "vaultstaticsecret", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultStaticSecret", "VaultStaticSecret": {"name":"standby-storage-creds","namespace":"postgres"
}, "namespace": "postgres", "name": "standby-storage-creds", "reconcileID": "7db8816c-e016-422a-9926-89cd1675cae7", "error": "VaultStaticSecret.secrets.hashicorp.com \"standby-storage-creds\" is invalid: status.lastGeneration: Required value"}

Expected behavior
All VaultStaticSecret are upgraded

Environment

• Kubernetes version: 1.22.2
  • Distribution or cloud vendor (OpenShift, EKS, GKE, AKS, etc.): onprem
  • Other configuration options or runtime services (istio, etc.):
• vault-secrets-operator version: v0.5.0

Additional context
Add any other context about the problem here.

[Ben Ash]

Hi @sebglon, thanks for reporting this issue. The status.lastGeneration field should have been updated after VSO was restarted during the upgrade. We will take a look internally to see what the issue could be. In the meantime, would you be able to provide us with some more log details?

Thanks,

Ben

[Sébastien GLON]

maybe an issue on our upgrade process; CRD was not updated before pods.
After upgrading CRDs all seems good

[Ben Ash]

Ah, could have been a transient issue then.

[Sébastien GLON]

I have reproduced the same issue on another cluster after upgrading CRD before deployment

[Ben Ash]

I think it is possible to run into this during the upgrade. In the case where you have applied the new CRD schema and have not yet brought up the new version of VSO that is compatible with the schema upgrade. These errors will occur until VSO has been restarted.

[Ben Ash]

Of note, these errors will be enqueued for reconciliation, so things will eventually be made consistent.

[Sébastien GLON]

I have restart VSO and the same issue:
more logs:

manager 2024-02-16T15:33:25Z    ERROR    Reconciler error    {"controller": "vaultstaticsecret", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultStaticSecret", "VaultStaticSecret": {"name":"regcred","namespace":"caposc-system"}, "names
pace": "caposc-system", "name": "regcred", "reconcileID": "50e7deef-eef8-4472-a214-883a07bfe7df", "error": "VaultStaticSecret.secrets.hashicorp.com \"regcred\" is invalid: status.lastGeneration: Required value"}
manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
manager     /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:329
manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
manager     /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:266
manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
manager     /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:227

[Ben Ash]

Ok, we'll take a closer look to see what might be going on.

[Ben Ash]

@sebglon could you attach the output of the following command here:

kubectl get vaultstaticsecrets.secrets.hashicorp.com -n caposc-system regcred -o yaml

[Ben Ash]

Closing as a duplicate of #608

[Sébastien GLON]

Always the same issue with 0.5.1