bug: seldon go clientset doesn't return correct status for updated deployment
brightsparc opened this issue · comments
I have noticed an issue whereby I can create a seldon deployment using the go client, and query the get command to return the status, it correctly transitions from Creating to Available.
But when I am updating a deployment I am always seeing the resulting status as being from the created action (ie Avaialble).
result, err = clientset.CoreV1().Secrets(secret.ObjectMeta.Namespace).Update(ctx, secret, metav1.UpdateOptions{})
dep, err := clientset.MachinelearningV1().SeldonDeployments(namespace).Get(ctx, deploymentName, metav1.GetOptions{ResourceVersion: result.ResourceVersion})
dep.Status.State == machinelearningv1.StatusStateAvailable
I've attempted to pass in the ResourceVersion in the options to the Get method, but this doesn't seem to help.
When I run the kubectl command I can see that in fact the update is still in the Creating state.
$ k get sdep golang-autoscaling -n predibase -o yaml
...
Deployment Status:
golang-autoscaling-default-0-triton-local:
Available Replicas: 1
Replicas: 2
Replicas: 2
State: Creating
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CreateService 5m37s seldon-controller-manager Created Service "golang-autoscaling-default-triton-local"
Normal CreateService 5m37s seldon-controller-manager Created Service "golang-autoscaling-default"
Normal CreateHPA 5m37s seldon-controller-manager Created HorizontalPodAutoscaler "golang-autoscaling-default-0-triton-local"
Normal CreateDeployment 5m37s seldon-controller-manager Created Deployment "golang-autoscaling-default-0-triton-local"
Normal UpdateDeployment 41s (x3 over 5m26s) seldon-controller-manager Updated Deployment "golang-autoscaling-default-0-triton-local"
Normal Updated 40s (x15 over 5m37s) seldon-controller-manager Updated SeldonDeployment "golang-autoscaling"
Warning UpdateFailed 40s (x3 over 5m37s) seldon-controller-manager Failed to update status for SeldonDeployment "golang-autoscaling": Operation cannot be fulfilled on seldondeployments.mach
inelearning.seldon.io "golang-autoscaling": the object has been modified; please apply your changes to the latest version and try again
Is there something I am missing here?