Possible old bug - getting reports of globalmount does not exist sporadically.
maprager opened this issue · comments
What happened:
Trying to mount a known drive occasionally fails with the message
I0122 16:14:54.715136 1 nodeserver.go:79] NodePublishVolume: mounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv//globalmount at /var/lib/kubelet/pods/65bba3c3-bdcb-4074-877a-ac23c575e5fc/volumes/kubernetes.iocsi//mount with mountOptions: [bind] volumeID(///
I0122 16:14:54.715165 1 mount_linux.go:220] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/pv//globalmount /var/lib/kubelet/pods/65bba3c3-bdcb-4074-877a-ac23c575e5fc/volumes/kubernetes.iocsi//mount)
E0122 16:14:54.716676 1 mount_linux.go:232] Mount failed: exit status 32
Mounting command: mount
Mounting arguments: -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/pv//globalmount /var/lib/kubelet/pods/65bba3c3-bdcb-4074-877a-ac23c575e5fc/volumes/kubernetes.io
Output: mount: /var/lib/kubelet/pods/65bba3c3-bdcb-4074-877a-ac23c575e5fc/volumes/kubernetes.io
What you expected to happen:
I expected the volume to be mounted - and the pod to start.
How to reproduce it:
Unknown - sometimes it works, and sometimes it doesn't
Anything else we need to know?:
- Other pods on the same node - seem to be able to access this pv - ( sometimes )
- K8s Version 1.21
- Baremetal K8s
- Driver info:
I1119 13:12:35.354327 1 smb.go:93]
DRIVER INFORMATION:
Build Date: "2023-09-11T23:25:57Z"
Compiler: gc
Driver Name: smb.csi.k8s.io
Driver Version: v1.13.0
Git Commit: ""
Go Version: go1.20.5
Platform: linux/amd64
Streaming logs below:
I1119 13:12:35.401233 1 mount_linux.go:284] Detected umount with safe 'not mounted' behavior
I1119 13:12:35.401925 1 driver.go:93] Enabling controller service capability: CREATE_DELETE_VOLUME
I1119 13:12:35.401945 1 driver.go:93] Enabling controller service capability: SINGLE_NODE_MULTI_WRITER
I1119 13:12:35.401949 1 driver.go:93] Enabling controller service capability: CLONE_VOLUME
I1119 13:12:35.401968 1 driver.go:112] Enabling volume access mode: SINGLE_NODE_WRITER
I1119 13:12:35.401973 1 driver.go:112] Enabling volume access mode: SINGLE_NODE_READER_ONLY
I1119 13:12:35.401976 1 driver.go:112] Enabling volume access mode: SINGLE_NODE_SINGLE_WRITER
I1119 13:12:35.401979 1 driver.go:112] Enabling volume access mode: SINGLE_NODE_MULTI_WRITER
I1119 13:12:35.401982 1 driver.go:112] Enabling volume access mode: MULTI_NODE_READER_ONLY
I1119 13:12:35.401987 1 driver.go:112] Enabling volume access mode: MULTI_NODE_SINGLE_WRITER
I1119 13:12:35.401990 1 driver.go:112] Enabling volume access mode: MULTI_NODE_MULTI_WRITER
I1119 13:12:35.401994 1 driver.go:103] Enabling node service capability: STAGE_UNSTAGE_VOLUME
I1119 13:12:35.401999 1 driver.go:103] Enabling node service capability: SINGLE_NODE_MULTI_WRITER
I1119 13:12:35.402001 1 driver.go:103] Enabling node service capability: VOLUME_MOUNT_GROUP
I1119 13:12:35.402005 1 driver.go:103] Enabling node service capability: GET_VOLUME_STATS
I1119 13:12:35.403111 1 server.go:118] Listening for connections on address: &net.UnixAddr{Name:"//csi/csi.sock", Net:"unix"}
I1119 13:12:35.969227 1 utils.go:76] GRPC call: /csi.v1.Identity/GetPluginInfo
I1119 13:12:35.969256 1 utils.go:77] GRPC request: {}
I1119 13:12:35.975036 1 utils.go:83] GRPC response: {"name":"smb.csi.k8s.io","vendor_version":"v1.13.0"}
I1119 13:12:36.063644 1 utils.go:76] GRPC call: /csi.v1.Identity/GetPluginInfo
I1119 13:12:36.063671 1 utils.go:77] GRPC request: {}
I1119 13:12:36.063755 1 utils.go:83] GRPC response: {"name":"smb.csi.k8s.io","vendor_version":"v1.13.0"}
I1119 13:12:36.599218 1 utils.go:76] GRPC call: /csi.v1.Node/NodeGetInfo
I1119 13:12:36.599236 1 utils.go:77] GRPC request: {}
I1119 13:12:36.599285 1 utils.go:83] GRPC response: {"node_id":"xxxxx"}
I1119 13:21:23.734807 1 utils.go:76] GRPC call: /csi.v1.Node/NodeStageVolume
please note: if I create the globablmount directory in the node manually - the mount continues, and the pod works -
kubectl exec -n kube-system -it csi-smb-node-v557q -c smb -- mkdir -p /var/lib/kubelet/plugins/kubernetes.io/csi/pv//globalmount
I found this solution here: #302
I'm also getting that error sporadically.
As @maprager mentioned, sometimes it works and sometimes it doesn't.
what's the volumeHandle value of your PVs? could you make sure there is no conflict of those volumeHandle values in all of your PVs?