operator attempts UPDATE on service even tho there was no change
sergeykuperman opened this issue · comments
Hello, we recently started having issue with one of our serviceinstances:
operator seems to try to update an instance even tho there was no change in spec.
the service in question (auditlog) does not support update, which results in permanent error in the status.
we are currently using version
sap/sap-btp-service-operator/controllerv0.5.8@sha256:61434048c3fe7d09658bcaac43bb4ba4297c7e6eb25d248078f43021219658b0
apiVersion: services.cloud.sap.com/v1
kind: ServiceInstance
metadata:
annotations:
helm.sh/hook-weight: "2"
meta.helm.sh/release-name: api-gw-center-auditlog
meta.helm.sh/release-namespace: auditlog
creationTimestamp: "2023-08-21T13:21:12Z"
finalizers:
- services.cloud.sap.com/sap-btp-finalizer
generation: 1
labels:
apigw.btp.sap.com/propagated_backup: "true"
app.kubernetes.io/managed-by: Helm
name: auditlog-oauth2
namespace: auditlog
resourceVersion: "1526106831"
uid: 5d12d4bc-379d-4c5e-a5c3-32442795862d
spec:
externalName: auditlog-oauth2
parameters:
xs-security:
oauth2-configuration:
credential-types:
- x509
grant-types:
- client_x509
- client_credentials
serviceOfferingName: auditlog
servicePlanName: oauth2
userInfo:
extra:
authentication.kubernetes.io/pod-name:
- operator-controller-manager-76d967bbdf-zf65z
authentication.kubernetes.io/pod-uid:
- <redacted>
groups:
- system:serviceaccounts
- system:serviceaccounts:operator-system
- system:authenticated
uid: <...>
username: system:serviceaccount:operator-system:operator
status:
conditions:
- lastTransitionTime: "2024-01-15T07:31:56Z"
message: 'ServiceInstance update failed: BrokerError:, Status: 400, Description:
Updating service is currently not supported operation. Please create a new service
with the desired updates and bind to it'
observedGeneration: 1
reason: UpdateFailed
status: "False"
type: Succeeded
- lastTransitionTime: "2023-08-21T13:21:14Z"
message: ""
reason: Provisioned
status: "True"
type: Ready
- lastTransitionTime: "2024-01-15T07:31:56Z"
message: 'ServiceInstance update failed: BrokerError:, Status: 400, Description:
Updating service is currently not supported operation. Please create a new service
with the desired updates and bind to it'
observedGeneration: 1
reason: UpdateFailed
status: "True"
type: Failed
hashedSpec: <...>
instanceID: <...>
observedGeneration: 1
ready: "True"
tags:
- auditlog
- audit log
- cf
is this expected behavior ?
Thank you for your assistance
Hi @sergeykuperman
This is not the expected behavior, update should not occur if spec is not changed.
There was a bug that caused an update to be triggered although spec was not changed and it was solved in v0.5.4
I was not able to reproduce your issue in v0.5.8, there is a log about the instance is in final state meaning it will not be updated
instance is in final state (generation: 1) {"serviceinstance": {"name":"audit","namespace":"default"...}
can you reproduce it?
i am not exactly sure which version of btp caused this issue, it could have been an older version like 0.5.4 and we only recently noticed this effect. if the current version does not reproduce it , i think the issue can be closed
Hi Colleagues,
we have the same problem with btp-service-operator and audit log service.
The problem is that btp-service-operator is setting/updating an userInfo field that can be changed by e.g. helm upgrade (in case ServiceInstance or ServiceBinding objects are rendered by helm chart).
This will trigger Update state of ServiceInstance object which is unsupported by audit log service.
There is mail thread with details which I'm attaching to this issue..
I will include @kerenlahav to the discussion.
I would suggest to re-open this issue until the discussion is settled.
Best regards
Daniel.
Re_ Audit Log service instance UpdateFailed.zip
Hi @dafe-corporate-sap
you didn't specify the operator's version, there was a bug about this issue until v0.5.4.
Hi @kerenlahav ,
we are using version: sap-btp-service-operator/controller:v0.5.3.
Are you suggesting us to upgrade to 0.6.0 and recheck?
Best regards
Daniel.
You can.
The bug that was fixed in v0.5.4 was an update request sent though the instance was not changed.
anyway this error does not affect the instance usability.
Hi @kerenlahav ,
this is a followup of issue by @dafe-corporate-sap
We updated the btp operator to v0.6.0 and the the service instance which is in Update failed is not getting updated.
Last Transition Time: 2024-02-12T12:04:32Z │ │ Message: ServiceInstance update failed: Status: 400; ErrorMessage: <nil>; Description: Updating service is currently not supported operation. Please create a new │ │ service with the desired updates and bind to it; ResponseError: <nil>
the btp operate did not try to reconcile this serviceistance object.
I would suggest to re-open this issue until the discussion is settled.
Hi @Fasee-khan
This is not a bug, it is the expected behavior, the status represents the last operation which was "update" that failed.
This error does not have any effect on the instance's usability.
In case of audit log instance there is nothing you can do to remove this status since the audit log broker does not support update and the last operation status will remain "update failed".
In version v0.6.0 instances are updated only when spec changed, you can test it with a new instance of audit log.
Thanks,
Keren.
Hi @kerenlahav,
we did another test with a new instance of audit log and applied the helm chart and it did not change the status of the service instances and they remain in the "created" state. This is evident that the issue is not occurring any more.
thanks , Fasee
Hi @kerenlahav,
there is a use case where btp-service-operator attempts to update audit-log service even though the user didn't intent to change anything.
Here are details:
kubectl create -f /tmp/auditlog.yaml
with
apiVersion: services.cloud.sap.com/v1
kind: ServiceInstance
metadata:
name: auditlog-test
namespace: unified-provisioning-poc
spec:
serviceOfferingName: auditlog
servicePlanName: oauth2
2. Replace it (identically) with a different user:
k replace -f /tmp/auditlog.yaml --as system:serviceaccount:unified-provisioning-poc:auditlog-test
then the instance end up in:
status:
conditions:
- lastTransitionTime: "2024-02-16T10:34:27Z"
message: 'BrokerError:, Status: 400, Description: Updating service is currently
not supported operation. Please create a new service with the desired updates
and bind to it'
observedGeneration: 2
reason: UpdateInProgress
status: "False"
type: Succeeded
- lastTransitionTime: "2024-02-16T10:34:13Z"
message: ""
reason: Provisioned
status: "True"
type: Ready
Is it really necessary that the userInfo field is being stored in Spec of the ServiceInstance object?
Hi @dafe-corporate-sap
It is not a valid scenario to change manually the UserInfo, UserInfo is updated by the operator when the spec changes.
I will prepare a fix to block user updates on UserInfo.
Thanks
fixed in #399