vr object remains on "from" cluster after failover, non-zero deletion timestamp but replication.storage.openshift.io finalizer remains, rpc error: code = InvalidArgument desc = secondary image status is up=true and state=unknown

[hatfieldbrian]

test:

1. rbd mirroring from cluster1 to hub
2. failover to hub: successful, but vr remains on cluster1

container image identifiers:

$ kubectl --context cluster1 describe po -nrook-ceph csi-rbdplugin-provisioner-7f8f7d4bf4-l7jl9

  volume-replication:
    Image:         quay.io/csiaddons/volumereplication-operator:latest
    Image ID:      docker-pullable://quay.io/csiaddons/volumereplication-operator@sha256:9c53ce8e3de3348dff5305fce4ff62eeaf02988855bb7bac246

  csi-rbdplugin:
    Image:         quay.io/cephcsi/cephcsi:canary
    Image ID:      docker-pullable://quay.io/cephcsi/cephcsi@sha256:70e019ad56bd409f6ab46748a64de331ee1b87b3aedc68d50689dda3bede321a

cluster1 vr log:

$ kubectl --context cluster1 -nrook-ceph logs deploy/csi-rbdplugin-provisioner volume-replication

vr-log-cluster1.txt

cluster1 vr log corresponding to its deletionTimestamp: "2021-09-29T22:46:51Z":

{"level":"info","timestamp":"2021-09-29T22:41:59.270Z","logger":"controllers.VolumeReplication","caller":"controller/controller.go:298","msg":"volume is marked primary","Request.Name":"busybox-pvc","Request.Namespace":"busybox-sample"}
{"level":"info","timestamp":"2021-09-29T22:46:33.212Z","logger":"controllers.VolumeReplication","caller":"controller/controller.go:298","msg":"volume handle","Request.Name":"busybox-pvc","Request.Namespace":"busybox-sample","VolumeHandleName":"0001-0009-rook-ceph-0000000000000002-741c9bcc-2176-11ec-99a4-0242ac11000a"}
{"level":"info","timestamp":"2021-09-29T22:46:33.988Z","logger":"controllers.VolumeReplication","caller":"controller/controller.go:298","msg":"volume is not ready to use","Request.Name":"busybox-pvc","Request.Namespace":"busybox-sample"}
{"level":"info","timestamp":"2021-09-29T22:46:49.009Z","logger":"controllers.VolumeReplication","caller":"controller/controller.go:298","msg":"volume handle","Request.Name":"busybox-pvc","Request.Namespace":"busybox-sample","VolumeHandleName":"0001-0009-rook-ceph-0000000000000002-741c9bcc-2176-11ec-99a4-0242ac11000a"}
{"level":"info","timestamp":"2021-09-29T22:46:49.254Z","logger":"controllers.VolumeReplication","caller":"controller/controller.go:298","msg":"volume is marked secondary","Request.Name":"busybox-pvc","Request.Namespace":"busybox-sample"}
{"level":"info","timestamp":"2021-09-29T22:46:51.375Z","logger":"controllers.VolumeReplication","caller":"controller/controller.go:298","msg":"volume handle","Request.Name":"busybox-pvc","Request.Namespace":"busybox-sample","VolumeHandleName":"0001-0009-rook-ceph-0000000000000002-741c9bcc-2176-11ec-99a4-0242ac11000a"}
{"level":"error","timestamp":"2021-09-29T22:46:51.520Z","logger":"controllers.VolumeReplication","caller":"controllers/volumereplication_controller.go:175","msg":"failed to disable volume replication","Request.Name":"busybox-pvc","Request.Namespace":"busybox-sample","error":"rpc error: code = InvalidArgument desc = secondary image status is up=true and state=unknown"}
{"level":"error","timestamp":"2021-09-29T22:46:51.520Z","logger":"controllers.VolumeReplication","caller":"controller/controller.go:298","msg":"failed to disable replication","Request.Name":"busybox-pvc","Request.Namespace":"busybox-sample","error":"rpc error: code = InvalidArgument desc = secondary image status is up=true and state=unknown"}
{"level":"error","timestamp":"2021-09-29T22:46:51.521Z","logger":"controller-runtime.manager.controller.volumereplication","caller":"controller/controller.go:253","msg":"Reconciler error","reconciler group":"replication.storage.openshift.io","reconciler kind":"VolumeReplication","name":"busybox-pvc","namespace":"busybox-sample","error":"rpc error: code = InvalidArgument desc = secondary image status is up=true and state=unknown"}
{"level":"info","timestamp":"2021-09-29T22:46:51.526Z","logger":"controllers.VolumeReplication","caller":"controller/controller.go:298","msg":"volume handle","Request.Name":"busybox-pvc","Request.Namespace":"busybox-sample","VolumeHandleName":"0001-0009-rook-ceph-0000000000000002-741c9bcc-2176-11ec-99a4-0242ac11000a"}
{"level":"error","timestamp":"2021-09-29T22:46:51.594Z","logger":"controllers.VolumeReplication","caller":"controllers/volumereplication_controller.go:175","msg":"failed to disable volume replication","Request.Name":"busybox-pvc","Request.Namespace":"busybox-sample","error":"rpc error: code = InvalidArgument desc = secondary image status is up=true and state=unknown"}
{"level":"error","timestamp":"2021-09-29T22:46:51.595Z","logger":"controllers.VolumeReplication","caller":"controller/controller.go:298","msg":"failed to disable replication","Request.Name":"busybox-pvc","Request.Namespace":"busybox-sample","error":"rpc error: code = InvalidArgument desc = secondary image status is up=true and state=unknown"}
{"level":"error","timestamp":"2021-09-29T22:46:51.595Z","logger":"controller-runtime.manager.controller.volumereplication","caller":"controller/controller.go:253","msg":"Reconciler error","reconciler group":"replication.storage.openshift.io","reconciler kind":"VolumeReplication","name":"busybox-pvc","namespace":"busybox-sample","error":"rpc error: code = InvalidArgument desc = secondary image status is up=true and state=unknown"}

[hatfieldbrian]

Updated volume-replication from
quay.io/csiaddons/volumereplication-operator@sha256:9c53ce8e3de3348dff5305fce4ff62eeaf02988855bb7bac246ec475e1879b48 to
quay.io/csiaddons/volumereplication-operator@sha256:f6659107601fdb55835855cfd59de66e311972894efbf60ac6e136fb90a13c64 and restarted csi-rbdplugin-provisioner deployment on each cluster, and unfinalized vr and pvc due to replication.storage.openshift.io finalizer issue is not present!