Crash-loop when a Service Binding remains with no Service Instance
adriil opened this issue · comments
Hi team,
When instantiating a Service Instance with a Service Binding in the same deployment, but the Service Instance provision failed and turned into a blocked status, one can be willing to delete the instance to be able to create a new one, while the Service Binding is still hanging. Unfortunately, this makes the btp-operator crash-loop with the following error (here the policysearch-canary-ai-core is the instance that doesn't exist anymore) :
2024-01-04T08:54:29Z INFO controllers.ServiceBinding getting service instance named policysearch-canary-ai-core in namespace pss for binding policysearch-canary-api-ai-core-binding in namespace pss
{"name":"policysearch-canary-api-ai-core-binding","namespace":"pss"}, "correlation_id": "3c453ac1-6ec3-49b9-8ae2-53c2e14baa86", "policysearch-canary-api-ai-core-binding": "pss"}
2024-01-04T08:54:29Z INFO Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference {"controller": "servicebinding", "controllerGroup": "services.cloud.sap.com", "controllerKind": "ServiceBinding", "ServiceBinding": {"name":"policysearch-canary-api-ai-core-binding","namespace":"pss"}, "namespace": "pss", "name": "policysearch-canary-api-ai-core-binding", "reconcileID": "c2146269-ec7e-4aba-b7aa-a676f7347090"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1a0 pc=0x14a2ca4]
goroutine 108 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:115 +0x1e5
panic({0x1607de0?, 0x25f3ed0?})
/usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/SAP/sap-btp-service-operator/controllers.(*ServiceBindingReconciler).Reconcile(0xc0000be378, {0x1a59d58, 0xc0000a6f60}, {{{0xc000013a1c, 0x3}, {0xc000049740, 0x27}}})
/workspace/controllers/servicebinding_controller.go:110 +0x664
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1a59d58?, {0x1a59d58?, 0xc0000a6f60?}, {{{0xc000013a1c?, 0x155ee20?}, {0xc000049740?, 0x1a4b950?}}})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000293e00, {0x1a59d90, 0xc000000820}, {0x167c900?, 0xc0000a8240?})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314 +0x368
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000293e00, {0x1a59d90, 0xc000000820})
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265 +0x1c9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 56
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:222 +0x565
Since the btp-operator is not healthy anymore, even deleting the remaining Service Binding is impossible. Moreover, further app deployments fail as long as they contain resources managed by operator, even without any change. So the user is blocked and only has two solutions : uninstall btp-operator and potentially break their app, or build a new cluster from scratch.
I think this is fixed by #386
Confirmed the issue is fixed after deployment of version 0.5.10