Catalog controller pod crashes in 0.3.0

Question

Catalog controller pod crashes in 0.3.0

addprs opened this issue 4 years ago · comments

Bug Report

Catalog controller pod crashes with panic error
What happened:
Tried to create service instances and servicebindings which runs into pod crash, eventually after 5 to 6 restarts everything works fine.

NAME                                                                           READY   STATUS    RESTARTS   AGE
catalog-catalog-controller-manager-95cc6fdc8-p6lt5   1/1     Running   6         10h

we see the pod restart happens around 6 times as above.
This works fine with 0.3.0-beta.2 version of catalog.

What you expected to happen:
Should be a clean run for provisioning. But we see the below error.

I0624 16:40:57.966721       1 controller_instance.go:3008] ServiceBinding testproject-1/testproject-1 triggered to reconciliation
E0624 16:40:57.966838       1 runtime.go:78] Observed a panic: "assignment to entry in nil map" (assignment to entry in nil map)
goroutine 275 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x197e3c0, 0x1e81a40)
        /go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/runtime/runtime.go:74 +0xa3
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
        /go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/runtime/runtime.go:48 +0x82
panic(0x197e3c0, 0x1e81a40)
        /usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/kubernetes-sigs/service-catalog/pkg/controller.(*controller).triggerServiceBindingReconciliation(0xc000508900, 0xc000ef6900)
        /go/src/github.com/kubernetes-sigs/service-catalog/pkg/controller/controller_instance.go:3010 +0x21e
github.com/kubernetes-sigs/service-catalog/pkg/controller.(*controller).processProvisionSuccess(0xc000508900, 0xc000ef6900, 0x0, 0x0, 0x0)
        /go/src/github.com/kubernetes-sigs/service-catalog/pkg/controller/controller_instance.go:2708 +0x1bd
github.com/kubernetes-sigs/service-catalog/pkg/controller.(*controller).reconcileServiceInstanceAdd(0xc000508900, 0xc000ef6000, 0x40a000, 0x0)
        /go/src/github.com/kubernetes-sigs/service-catalog/pkg/controller/controller_instance.go:668 +0xa91
github.com/kubernetes-sigs/service-catalog/pkg/controller.(*controller).reconcileServiceInstance(0xc000508900, 0xc000ef6000, 0x0, 0x0)
        /go/src/github.com/kubernetes-sigs/service-catalog/pkg/controller/controller_instance.go:336 +0x347
github.com/kubernetes-sigs/service-catalog/pkg/controller.(*controller).reconcileServiceInstanceKey(0xc000508900, 0xc000f14f00, 0x21, 0xc000f1e530, 0x0)
        /go/src/github.com/kubernetes-sigs/service-catalog/pkg/controller/controller_instance.go:306 +0x2f4
github.com/kubernetes-sigs/service-catalog/pkg/controller.worker.func1.1(0x1f142a0, 0xc000596fc0, 0xc0009ba4c0, 0x1, 0xf, 0x1be83a6, 0xf, 0x1000772500)
        /go/src/github.com/kubernetes-sigs/service-catalog/pkg/controller/controller.go:394 +0xff
github.com/kubernetes-sigs/service-catalog/pkg/controller.worker.func1()
        /go/src/github.com/kubernetes-sigs/service-catalog/pkg/controller/controller.go:412 +0x8a
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00097ef40)
        /go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:155 +0x5e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00097ef40, 0x1e9e3c0, 0xc000950ea0, 0xc0000f4001, 0x0)
        /go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:156 +0xa3
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00097ef40, 0x3b9aca00, 0x0, 0x1, 0x0)
        /go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:133 +0xe2
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:90
github.com/kubernetes-sigs/service-catalog/pkg/controller.createWorker.func1(0x1f142a0, 0xc000596fc0, 0x1be83a6, 0xf, 0xf, 0x1, 0xc0009ba4c0, 0x0, 0xc00046ce60)
        /go/src/github.com/kubernetes-sigs/service-catalog/pkg/controller/controller.go:310 +0x9f
created by github.com/kubernetes-sigs/service-catalog/pkg/controller.createWorker
        /go/src/github.com/kubernetes-sigs/service-catalog/pkg/controller/controller.go:309 +0xbe
panic: assignment to entry in nil map [recovered]
        panic: assignment to entry in nil map

goroutine 275 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
        /go/pkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/runtime/runtime.go:55 +0x105
panic(0x197e3c0, 0x1e81a40)

How to reproduce it (as minimally and precisely as possible):
Kubernetes version: v1.15, v1.18
catalog : 0.3.0
Note: This did not occur in 0.3.0-beta.2 version
Install service catalog with helm and try to create serviceinstance/ bindings

Anything else we need to know?:

Environment:

Kubernetes version (use kubectl version): 1.15 to 1.18
service-catalog version: 0.3.0
Cloud provider or hardware configuration:
Do you have api aggregation enabled?
- Do you see the configmap in kube-system? yes
- Does it have all the necessary fields?
  - kubectl get cm -n kube-system extension-apiserver-authentication -o yaml and look for requestheader-XXX fields
Install tools:
- Did you use helm? What were the helm arguments? Did you --set any extra values?
  yes helm v3.2.0
Are you trying to use ALPHA features? Did you enable them?

priyanka adama · Answer 1 · Fri Aug 14 2020 10:43:27 GMT+0800 (China Standard Time)

Can I please know if there is any update on the above issue?

Christophe Balczunas · Answer 2 · Wed Sep 09 2020 22:56:03 GMT+0800 (China Standard Time)

Bumping this issue. We're not able to make sense of the panic stack.

priyanka adama · Answer 3 · Tue Sep 22 2020 07:22:19 GMT+0800 (China Standard Time)

The panic error caused above is because of the assignment into the map after doing deep copy, this may be because of the line added here https://github.com/kubernetes-sigs/service-catalog/blob/v0.3.0/pkg/controller/controller_instance.go#L3010 , which assumes there is a map, where key can be inserted.

If the serviceinstanes, servicebinding don't have the annotation we run into the problem, Not sure if this is bug by the service catalog 0.3.0. But adding some dummy annotations helped fix the problem.

lazywhite · Answer 4 · Wed Dec 16 2020 17:12:24 GMT+0800 (China Standard Time)

The panic error caused above is because of the assignment into the map after doing deep copy, this may be because of the line added here https://github.com/kubernetes-sigs/service-catalog/blob/v0.3.0/pkg/controller/controller_instance.go#L3010 , which assumes there is a map, where key can be inserted.

If the serviceinstanes, servicebinding don't have the annotation we run into the problem, Not sure if this is bug by the service catalog 0.3.0. But adding some dummy annotations helped fix the problem.

I've try this, problem solved!!! thanks.

Juldrixx · Answer 5 · Wed Feb 24 2021 22:24:05 GMT+0800 (China Standard Time)

Hi, I've got the same issue. Is it possible to add a test to check whether the annotation field is empty or not ?