Unable to unprovisioning a binding that is 'gone'

Question

Unable to unprovisioning a binding that is 'gone'

janssk1 opened this issue 4 years ago · comments

Bug Report

What happened:
I'm having a broker that provisions/unprovisions DB schemas. When a DB schema got unprovisioned out-of-band and the service-catalog tries to unprovision it, the broker responds with a 410 (Gone) with empty body. The service catalog does not like that. It expect the response to include a json body and fails while trying to parse it: ' unexpected end of JSON input'

What you expected to happen:
Service catalog should see that the binding no longer exists an clean it, without trying to parse the empty body

How to reproduce it (as minimally and precisely as possible):
See above

Anything else we need to know?:
AFAIK, it seems to be an issue with the OSB go client library. Current workaround is have is to change the broker to return 200 OK in case the binding no longer exists, instead of the spec compliant 410 GONE.

Environment:

Kubernetes version (use kubectl version):
1.18
service-catalog version:
v0.3.1

Detailed logs

I1211 13:57:16.158334 I1211 13:57:16.158443 I1211 13:57:19.316502 I1211 13:57:19.316533 I1211 13:57:19.316563 I1211 13:57:19.316616 I1211 13:57:19.316702 I1211 13:57:19.319655 I1211 13:57:19.319680 I1211 13:57:19.319685 I1211 13:57:19.319688 I1211 13:57:19.319692 I1211 13:57:19.319698 I1211 13:57:19.319702 I1211 13:57:19.319707 I1211 13:57:19.319726 I1211 13:57:19.319764 I1211 13:57:19.319942 I1211 13:57:19.348102 I1211 13:57:19.348132 I1211 13:57:19.348141 I1211 13:57:19.348277 I1211 13:57:19.348357 I1211 13:57:19.348433 I1211 13:57:19.357604 I1211 13:57:19.357626 I1211 13:57:19.357630 I1211 13:57:19.357634 I1211 13:57:19.357637 I1211 13:57:19.357640 I1211 13:57:19.357642 I1211 13:57:19.357645 I1211 13:57:19.357648 I1211 13:57:19.357695 I1211 13:57:19.357884 I1211 13:57:19.357897 1 request.go:1068] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"cluster-info","namespace":"catalog","selfLink":"/api/v1/namespaces/catalog/configmaps/cluster-info","uid":"f6ce10fd-8ca1-44ae-ac68-21faee3ae78c","resourceVersion":"65304728","creationTimestamp":"2020-11-20T10:50:21Z","managedFields":[{"manager":"service-catalog","operation":"Update","apiVersion":"v1","time":"2020-11-20T10:50:21Z","fieldsType":"FieldsV1","fieldsV1":{"f:data":{".":{},"f:id":{}}}}]},"data":{"id":"487218b4-77be-4d05-b7ec-ce6f0f6d36f5"}}
1 controller.go:370] cluster ID monitor loop exit
1 controller_binding.go:172] ServiceBinding "koen/mysql-wage-determination-service" v85826610: beginning to process resourceVersion: 85826610
1 controller_binding.go:388] ServiceBinding "koen/mysql-wage-determination-service" v85826610: Processing Delete
1 controller_binding.go:660] ServiceBinding "koen/mysql-wage-determination-service" v85826610: Deleting Secret "koen/mysql-wage-determination-service"
1 request.go:1068] Request Body: {"kind":"DeleteOptions","apiVersion":"v1"}
1 round_trippers.go:423] curl -k -v -XDELETE -H "Content-Type: application/json" -H "Accept: application/json, /" -H "User-Agent: service-catalog/v0.0.0 (linux/amd64) kubernetes/$Format/service-catalog-controller-manager" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImNVQjAtNnpMYUJUMDU0X1kxNzNUeUtXNHByZkJkZzhWemZGZG52U29hSVUifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjYXRhbG9nIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InNlcnZpY2UtY2F0YWxvZy1jb250cm9sbGVyLW1hbmFnZXItdG9rZW4tN3RuZnEiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoic2VydmljZS1jYXRhbG9nLWNvbnRyb2xsZXItbWFuYWdlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImE5NTAwNWYyLTI5MWEtNDIxMC05MjQwLTQ2Yzc2MjI4MGU1YyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpjYXRhbG9nOnNlcnZpY2UtY2F0YWxvZy1jb250cm9sbGVyLW1hbmFnZXIifQ.G4YP_zIrDmFSa_S67gIautgQQA86DwDFNrFsS-VlGEA6GhLv6b4B2h7-_P386vbX4d2gl6cDX_jxvu5tL7ldw9pcwAqwUyz3V4CloCk7WSWSIHD6UsLz1G72NO1R-B0qeGIy5MjEUCB6Sl-OeT_XbALZisseB3LgFZiekCajtUBf91eO1l-abk4TXgTtCl8P1mhV4KpX2LzADfvgLUFq_TS2U8tLV93h0vxi2MicSD3Xgqq7JelId2nH9WgVUcNlSDKpT6WTjMB8ce28YW9wUsJSLTIEkoCXQDDK0dP2-Xg8O-PyJvaMbTh6f1ESrJawfFUYnT2vpbF4bcit8IC3yw" 'https://172.30.0.1:443/api/v1/namespaces/koen/secrets/mysql-wage-determination-service'
1 round_trippers.go:443] DELETE https://172.30.0.1:443/api/v1/namespaces/koen/secrets/mysql-wage-determination-service 404 Not Found in 2 milliseconds
1 round_trippers.go:449] Response Headers:
1 round_trippers.go:452] Content-Length: 238
1 round_trippers.go:452] Date: Fri, 11 Dec 2020 13:57:19 GMT
1 round_trippers.go:452] Audit-Id: 653f52db-6aa0-41ae-bb18-7b12244d66c4
1 round_trippers.go:452] Cache-Control: no-cache, private
1 round_trippers.go:452] Content-Type: application/json
1 round_trippers.go:452] X-Kubernetes-Pf-Flowschema-Uid: 1ea23974-276a-416e-9c11-5f2a7ccb2c3f
1 round_trippers.go:452] X-Kubernetes-Pf-Prioritylevel-Uid: 932442f5-9ca5-4b6b-b72b-2f06f7937842
1 request.go:1068] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets "mysql-wage-determination-service" not found","reason":"NotFound","details":{"name":"mysql-wage-determination-service","kind":"secrets"},"code":404}
1 osbproxy.go:143] OSBClientProxy Unbind()
1 controller_binding.go:698] ServiceBinding "koen/mysql-wage-determination-service" v85826610: Error unbinding from ServiceInstance "koen/mysql-wage-determination-service" of ServiceClass (K8S: "koen/be.everesst.mysql" ExternalName: "mysql") at ServiceBroker "mysql-broker": Status: 410; ErrorMessage: ; Description: ; ResponseError: unexpected end of JSON input
1 controller_binding.go:699] ServiceBinding "koen/mysql-wage-determination-service" v85826610: Setting condition "Ready" to Unknown
1 controller_binding.go:748] ServiceBinding "koen/mysql-wage-determination-service" v85826610: Updating status
1 request.go:1068] Request Body: {"kind":"ServiceBinding","apiVersion":"servicecatalog.k8s.io/v1beta1","metadata":{"name":"mysql-wage-determination-service","namespace":"koen","selfLink":"/apis/servicecatalog.k8s.io/v1beta1/namespaces/koen/servicebindings/mysql-wage-determination-service","uid":"89bb59b4-aee4-4607-86c7-6cbf8b01d5f1","resourceVersion":"85826610","generation":2,"creationTimestamp":"2020-12-10T16:12:12Z","deletionTimestamp":"2020-12-11T13:22:21Z","deletionGracePeriodSeconds":0,"labels":{"app.kubernetes.io/managed-by":"Helm"},"annotations":{"meta.helm.sh/release-name":"wds","meta.helm.sh/release-namespace":"koen","reconciliationTriggered":"2020-12-10 16:12:13.080996608 +0000 UTC m=+1747318.839884697"},"finalizers":["kubernetes-incubator/service-catalog"],"managedFields":[{"manager":"Go-http-client","operation":"Update","apiVersion":"servicecatalog.k8s.io/v1beta1","time":"2020-12-10T16:12:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app.kubernetes.io/managed-by":{}}},"f:spec":{".":{},"f:instanceRef":{".":{},"f:name":{}}}}},{"manager":"service-catalog","operation":"Update","apiVersion":"servicecatalog.k8s.io/v1beta1","time":"2020-12-11T13:22:21Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:reconciliationTriggered":{}}},"f:status":{".":{},"f:asyncOpInProgress":{},"f:conditions":{},"f:currentOperation":{},"f:externalProperties":{".":{},"f:userInfo":{".":{},"f:extra":{".":{},"f:scopes.authorization.openshift.io":{}},"f:groups":{},"f:uid":{},"f:username":{}}},"f:lastConditionState":{},"f:operationStartTime":{},"f:orphanMitigationInProgress":{},"f:reconciledGeneration":{},"f:unbindStatus":{}}}}]},"spec":{"instanceRef":{"name":"mysql-wage-determination-service"},"secretName":"mysql-wage-determination-service","externalID":"73392012-fdcb-4355-8a0d-cdcf06cc937b","userInfo":{"username":"Koen.Janssens3@cegeka.com","uid":"f328dbf3-c9b0-4483-a36b-d6e329434f19","groups":["GL-Acerta-Everesst-RW","system:authenticated:oauth","system:authenticated"],"extra":{"scopes.authorization.openshift.io":["user:full"]}}},"status":{"conditions":[{"type":"Ready","status":"Unknown","lastTransitionTime":"2020-12-11T13:22:21Z","reason":"UnbindCallFailed","message":"Error unbinding from ServiceInstance "koen/mysql-wage-determination-service" of ServiceClass (K8S: "koen/be.everesst.mysql" ExternalName: "mysql") at ServiceBroker "mysql-broker": Status: 410; ErrorMessage: \u003cnil\u003e; Description: \u003cnil\u003e; ResponseError: unexpected end of JSON input"}],"asyncOpInProgress":false,"currentOperation":"Unbind","reconciledGeneration":1,"operationStartTime":"2020-12-11T13:22:21Z","externalProperties":{"userInfo":{"username":"Koen.Janssens3@cegeka.com","uid":"f328dbf3-c9b0-4483-a36b-d6e329434f19","groups":["GL-Acerta-Everesst-RW","system:authenticated:oauth","system:authenticated"],"extra":{"scopes.authorization.openshift.io":["user:full"]}}},"orphanMitigationInProgress":false,"unbindStatus":"Required","lastConditionState":"UnbindCallFailed"}}
1 event.go:278] Event(v1.ObjectReference{Kind:"ServiceBinding", Namespace:"koen", Name:"mysql-wage-determination-service", UID:"89bb59b4-aee4-4607-86c7-6cbf8b01d5f1", APIVersion:"servicecatalog.k8s.io/v1beta1", ResourceVersion:"85826610", FieldPath:""}): type: 'Warning' reason: 'UnbindCallFailed' Error unbinding from ServiceInstance "koen/mysql-wage-determination-service" of ServiceClass (K8S: "koen/be.everesst.mysql" ExternalName: "mysql") at ServiceBroker "mysql-broker": Status: 410; ErrorMessage: ; Description: ; ResponseError: unexpected end of JSON input
1 round_trippers.go:423] curl -k -v -XPUT -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImNVQjAtNnpMYUJUMDU0X1kxNzNUeUtXNHByZkJkZzhWemZGZG52U29hSVUifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjYXRhbG9nIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InNlcnZpY2UtY2F0YWxvZy1jb250cm9sbGVyLW1hbmFnZXItdG9rZW4tN3RuZnEiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoic2VydmljZS1jYXRhbG9nLWNvbnRyb2xsZXItbWFuYWdlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImE5NTAwNWYyLTI5MWEtNDIxMC05MjQwLTQ2Yzc2MjI4MGU1YyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpjYXRhbG9nOnNlcnZpY2UtY2F0YWxvZy1jb250cm9sbGVyLW1hbmFnZXIifQ.G4YP_zIrDmFSa_S67gIautgQQA86DwDFNrFsS-VlGEA6GhLv6b4B2h7-_P386vbX4d2gl6cDX_jxvu5tL7ldw9pcwAqwUyz3V4CloCk7WSWSIHD6UsLz1G72NO1R-B0qeGIy5MjEUCB6Sl-OeT_XbALZisseB3LgFZiekCajtUBf91eO1l-abk4TXgTtCl8P1mhV4KpX2LzADfvgLUFq_TS2U8tLV93h0vxi2MicSD3Xgqq7JelId2nH9WgVUcNlSDKpT6WTjMB8ce28YW9wUsJSLTIEkoCXQDDK0dP2-Xg8O-PyJvaMbTh6f1ESrJawfFUYnT2vpbF4bcit8IC3yw" -H "Accept: application/json, /" -H "Content-Type: application/json" -H "User-Agent: service-catalog/v0.0.0 (linux/amd64) kubernetes/$Format/service-catalog-controller-manager" 'https://172.30.0.1:443/apis/servicecatalog.k8s.io/v1beta1/namespaces/koen/servicebindings/mysql-wage-determination-service/status'
1 round_trippers.go:443] PUT https://172.30.0.1:443/apis/servicecatalog.k8s.io/v1beta1/namespaces/koen/servicebindings/mysql-wage-determination-service/status 200 OK in 9 milliseconds
1 round_trippers.go:449] Response Headers:
1 round_trippers.go:452] Date: Fri, 11 Dec 2020 13:57:19 GMT
1 round_trippers.go:452] Audit-Id: d92eae08-59ad-4741-b53b-f8477e5a09dc
1 round_trippers.go:452] Cache-Control: no-cache, private
1 round_trippers.go:452] Content-Type: application/json
1 round_trippers.go:452] X-Kubernetes-Pf-Flowschema-Uid: 1ea23974-276a-416e-9c11-5f2a7ccb2c3f
1 round_trippers.go:452] X-Kubernetes-Pf-Prioritylevel-Uid: 932442f5-9ca5-4b6b-b72b-2f06f7937842
1 round_trippers.go:452] Content-Length: 3113
1 request.go:1068] Response Body: {"apiVersion":"servicecatalog.k8s.io/v1beta1","kind":"ServiceBinding","metadata":{"annotations":{"meta.helm.sh/release-name":"wds","meta.helm.sh/release-namespace":"koen","reconciliationTriggered":"2020-12-10 16:12:13.080996608 +0000 UTC m=+1747318.839884697"},"creationTimestamp":"2020-12-10T16:12:12Z","deletionGracePeriodSeconds":0,"deletionTimestamp":"2020-12-11T13:22:21Z","finalizers":["kubernetes-incubator/service-catalog"],"generation":2,"labels":{"app.kubernetes.io/managed-by":"Helm"},"managedFields":[{"apiVersion":"servicecatalog.k8s.io/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app.kubernetes.io/managed-by":{}}},"f:spec":{".":{},"f:instanceRef":{".":{},"f:name":{}}}},"manager":"Go-http-client","operation":"Update","time":"2020-12-10T16:12:12Z"},{"apiVersion":"servicecatalog.k8s.io/v1beta1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:reconciliationTriggered":{}}},"f:status":{".":{},"f:asyncOpInProgress":{},"f:conditions":{},"f:currentOperation":{},"f:externalProperties":{".":{},"f:userInfo":{".":{},"f:extra":{".":{},"f:scopes.authorization.openshift.io":{}},"f:groups":{},"f:uid":{},"f:username":{}}},"f:lastConditionState":{},"f:operationStartTime":{},"f:orphanMitigationInProgress":{},"f:reconciledGeneration":{},"f:unbindStatus":{}}},"manager":"service-catalog","operation":"Update","time":"2020-12-11T13:22:21Z"}],"name":"mysql-wage-determination-service","namespace":"koen","resourceVersion":"85826610","selfLink":"/apis/servicecatalog.k8s.io/v1beta1/namespaces/koen/servicebindings/mysql-wage-determination-service/status","uid":"89bb59b4-aee4-4607-86c7-6cbf8b01d5f1"},"spec":{"externalID":"73392012-fdcb-4355-8a0d-cdcf06cc937b","instanceRef":{"name":"mysql-wage-determination-service"},"secretName":"mysql-wage-determination-service","userInfo":{"extra":{"scopes.authorization.openshift.io":["user:full"]},"groups":["GL-Acerta-Everesst-RW","system:authenticated:oauth","system:authenticated"],"uid":"f328dbf3-c9b0-4483-a36b-d6e329434f19","username":"Koen.Janssens3@cegeka.com"}},"status":{"asyncOpInProgress":false,"conditions":[{"lastTransitionTime":"2020-12-11T13:22:21Z","message":"Error unbinding from ServiceInstance "koen/mysql-wage-determination-service" of ServiceClass (K8S: "koen/be.everesst.mysql" ExternalName: "mysql") at ServiceBroker "mysql-broker": Status: 410; ErrorMessage: \u003cnil\u003e; Description: \u003cnil\u003e; ResponseError: unexpected end of JSON input","reason":"UnbindCallFailed","status":"Unknown","type":"Ready"}],"currentOperation":"Unbind","externalProperties":{"userInfo":{"extra":{"scopes.authorization.openshift.io":["user:full"]},"groups":["GL-Acerta-Everesst-RW","system:authenticated:oauth","system:authenticated"],"uid":"f328dbf3-c9b0-4483-a36b-d6e329434f19","username":"Koen.Janssens3@cegeka.com"}},"lastConditionState":"UnbindCallFailed","operationStartTime":"2020-12-11T13:22:21Z","orphanMitigationInProgress":false,"reconciledGeneration":1,"unbindStatus":"Required"}}
1 controller_binding.go:753] ServiceBinding "koen/mysql-wage-determination-service" v85826610: Updated status of resourceVersion: 85826610; got resourceVersion: 85826610
1 controller.go:404] Dropping ServiceBinding "koen/mysql-wage-determination-service" out of the queue: Error unbinding from ServiceInstance "koen/mysql-wage-determination-service" of ServiceClass (K8S: "koen/be.everesst.mysql" ExternalName: "mysql") at ServiceBroker "mysql-broker": Status: 410; ErrorMessage: ; Description: ; ResponseError: unexpected end of JSON input

rradillen commented 3 years ago

/reopen

fejta-bot · Answer 1 · Thu Mar 11 2021 23:20:07 GMT+0800 (China Standard Time)

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

fejta-bot · Answer 2 · Sat Apr 10 2021 23:59:59 GMT+0800 (China Standard Time)

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

fejta-bot · Answer 3 · Tue May 11 2021 00:15:56 GMT+0800 (China Standard Time)

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Kubernetes Prow Robot · Answer 4 · Tue May 11 2021 00:16:02 GMT+0800 (China Standard Time)

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

rradillen · Answer 5 · Wed Nov 03 2021 22:00:04 GMT+0800 (China Standard Time)

/reopen /remove-lifecycle rotten

rradillen · Answer 6 · Wed Nov 03 2021 22:02:36 GMT+0800 (China Standard Time)

We are still experiencing this issue
/sig sig-service-catalog

Kubernetes Prow Robot · Answer 7 · Wed Nov 03 2021 22:02:37 GMT+0800 (China Standard Time)

@rradillen: The label(s) sig/sig-service-catalog cannot be applied, because the repository doesn't have them.

In response to this:

We are still experiencing this issue
/sig sig-service-catalog

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

rradillen · Answer 8 · Wed Nov 03 2021 22:03:00 GMT+0800 (China Standard Time)

/sig service-catalog

Kubernetes Prow Robot · Answer 9 · Wed Nov 03 2021 23:46:20 GMT+0800 (China Standard Time)

@rradillen: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.