Potential non-idempotency issue in SMB CSI Driver on Windows resulting in access is denied error
tzifudzi opened this issue · comments
What happened:
I've encountered a scenario where NodePublishVolume seems to be called twice in a very short timespan (0.102s). The first call appears to be successful, but the subsequent call results in an error. It seems that due to the successful outcome of the initial call, a file system control lock is held, which then causes the second operation to fail with an "Access is denied" error. I validated this by using Procmon to identify that the reason for access denied is the file lock.
The following is a snippet of the logs describing the failure event. Note I have redacted some UIDs and replaced them with random replacements e.g. the pod name is ccccc-cccccccccccc-ccc-cccccc-ccccc-c-c-ccccc.
I1208 16:00:27.747297 1664 utils.go:76] GRPC call: /csi.v1.Node/NodePublishVolume
I1208 16:00:27.747297 1664 utils.go:77] GRPC request: {"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\smb.csi.k8s.io\\bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\\globalmount","target_path":"c:\\var\\lib\\kubelet\\pods\\aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa\\volumes\\kubernetes.io~csi\\something-pv-csi\\mount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["dir_mode=0777","file_mode=0777","vers=3.0"]}},"access_mode":{"mode":5}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"ccccc-cccccccccccc-ccc-cccccc-ccccc-c-c-ccccc","csi.storage.k8s.io/pod.namespace":"something-namespace","csi.storage.k8s.io/pod.uid":"aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa","csi.storage.k8s.io/serviceAccount.name":"default","source":"\\\\10.30.30.30\\share\\env\\some-folder"},"volume_id":"something-vh-csi"}
I1208 16:00:27.747297 1664 safe_mounter_windows.go:230] IsLikelyNotMountPoint: c:\var\lib\kubelet\pods\aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa\volumes\kubernetes.io~csi\something-pv-csi\mount
I1208 16:00:27.747297 1664 safe_mounter_windows.go:301] Exists path: c:\var\lib\kubelet\pods\aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa\volumes\kubernetes.io~csi\something-pv-csi\mount
I1208 16:00:27.748731 1664 safe_mounter_windows.go:301] Exists path: c:\var\lib\kubelet\pods\aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa\volumes\kubernetes.io~csi\something-pv-csi\mount
I1208 16:00:27.749051 1664 smb_common_windows.go:63] Removing path: c:\var\lib\kubelet\pods\aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa\volumes\kubernetes.io~csi\something-pv-csi\mount
I1208 16:00:27.749121 1664 safe_mounter_windows.go:199] Remove directory: c:\var\lib\kubelet\pods\aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa\volumes\kubernetes.io~csi\something-pv-csi\mount
I1208 16:00:27.749206 1664 nodeserver.go:79] NodePublishVolume: mounting \var\lib\kubelet\plugins\kubernetes.io\csi\smb.csi.k8s.io\bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\globalmount at c:\var\lib\kubelet\pods\aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa\volumes\kubernetes.io~csi\something-pv-csi\mount with mountOptions: [bind] volumeID(something-vh-csi)
I1208 16:00:27.749206 1664 safe_mounter_windows.go:175] Mount: old name: \var\lib\kubelet\plugins\kubernetes.io\csi\smb.csi.k8s.io\bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\globalmount. new name: c:\var\lib\kubelet\pods\aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa\volumes\kubernetes.io~csi\something-pv-csi\mount
I1208 16:00:27.751407 1664 nodeserver.go:86] NodePublishVolume: mount \var\lib\kubelet\plugins\kubernetes.io\csi\smb.csi.k8s.io\bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\globalmount at c:\var\lib\kubelet\pods\aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa\volumes\kubernetes.io~csi\something-pv-csi\mount volumeID(something-vh-csi) successfully
I1208 16:00:27.751407 1664 utils.go:83] GRPC response: {}
I1208 16:00:27.851468 1664 utils.go:76] GRPC call: /csi.v1.Node/NodePublishVolume
I1208 16:00:27.851468 1664 utils.go:77] GRPC request: {"staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\smb.csi.k8s.io\\bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\\globalmount","target_path":"c:\\var\\lib\\kubelet\\pods\\aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa\\volumes\\kubernetes.io~csi\\something-pv-csi\\mount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["dir_mode=0777","file_mode=0777","vers=3.0"]}},"access_mode":{"mode":5}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"ccccc-cccccccccccc-ccc-cccccc-ccccc-c-c-ccccc","csi.storage.k8s.io/pod.namespace":"something-namespace","csi.storage.k8s.io/pod.uid":"aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa","csi.storage.k8s.io/serviceAccount.name":"default","source":"\\\\10.30.30.30\\share\\env\\some-folder"},"volume_id":"something-vh-csi"}
I1208 16:00:27.851468 1664 safe_mounter_windows.go:230] IsLikelyNotMountPoint: c:\var\lib\kubelet\pods\aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa\volumes\kubernetes.io~csi\something-pv-csi\mount
I1208 16:00:27.851468 1664 safe_mounter_windows.go:301] Exists path: c:\var\lib\kubelet\pods\aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa\volumes\kubernetes.io~csi\something-pv-csi\mount
W1208 16:00:27.857327 1664 nodeserver.go:400] ReadDir c:\var\lib\kubelet\pods\aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa\volumes\kubernetes.io~csi\something-pv-csi\mount failed with open c:\var\lib\kubelet\pods\aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa\volumes\kubernetes.io~csi\something-pv-csi\mount: Access is denied., unmount this directory
I1208 16:00:27.857327 1664 safe_mounter_windows.go:213] Unmount: c:\var\lib\kubelet\pods\aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa\volumes\kubernetes.io~csi\something-pv-csi\mount
I1208 16:00:27.857327 1664 safe_mounter_windows.go:199] Remove directory: c:\var\lib\kubelet\pods\aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa\volumes\kubernetes.io~csi\something-pv-csi\mount
E1208 16:00:27.858089 1664 utils.go:81] GRPC error: rpc error: code = Internal desc = Could not mount target "c:\\var\\lib\\kubelet\\pods\\aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa\\volumes\\kubernetes.io~csi\\something-pv-csi\\mount": open c:\var\lib\kubelet\pods\aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa\volumes\kubernetes.io~csi\something-pv-csi\mount: Access is denied.
What I expected to happen:
According to the CSI specification, the NodePublishVolume operation must be idempotent. As stated,
This operation MUST be idempotent. If the volume corresponding to the volume_id has already been published at the specified target_path, and is compatible with the specified volume_capability and readonly flag, the Plugin MUST reply 0 OK.
Therefore, I would expect that if NodePublishVolume is called multiple times with identical parameters within a short timespan, it should not result in the access is denied error. Instead, the operation should confirm that the volume is published without any such adverse effects.
How to reproduce it:
I'm currently unable to reliably reproduce this issue. The error occurs very sporadically. It is a tall ask, but I believe that it might be possible to simulate this behavior in end to end or integration tests by performing simultaneous gRPC calls for the same pod, volume, volume capability, and staging_target_path within a very short interval, such as 5 microseconds.
Anything else we need to know?:
This issue is occurring in a cloud environment whereby EKS Windows AMIs are being used with this project to access SMB file shares in Amazon FSx. I checked the Active Directory logs and ruled out access to the SMB file shares being the source of access is denied.
I understand that the lack of reliable reproduction steps makes it challenging to effectively troubleshoot this issue. For the interim, I'm looking for confirmation or insights regarding my theory, regarding whether the CSI operation called twice within a short timespan should not return errors.
Environment details:
- CSI Driver version: v1.12.0
- Kubernetes version (use kubectl version): v1.26.10
- OS (e.g., from /etc/os-release): Windows Server Datacenter 2022 amd64
- CSI Proxy Version: v1.1.3
I took a further look at the code, and I'd like to add an additional observation.
Based on the current implementation, the NodeStageVolume and NodeUnstageVolume methods (in nodeserver.go) utilize a locking mechanism to ensure that only one operation occurs at a time for a given volume. Taking a closer look at the NodePublishVolume method, given the two operations are being carried out quite closely in time, could this issue possibly be mitigated by utilizing a thread-safe mutex lock for the NodePublishVolume operation?
there is retry if NodePublishVolume failed at first time, will retry work?
@andyzhangx Because of the difficulty of reproducing the issue, its difficult to say for certain but I will try look deeper into the logs to answer if a retry is attempted and whether its successful.
However for now what I can say, is that it appears a retry is not attempted. The SMB CSI driver tries to recover by unmounting and remounting, but that fails because of an access denied error and the binding of the volume for the pod fails and the entire container creation fails.
@tzifudzi the two NodePublishVolume operations are for the same pod?
in 16:00:27.749206, the symlink of c:\var\lib\kubelet\pods\aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa\volumes\kubernetes.io~csi\something-pv-csi\mount is created, while in 16:00:27.857327, it returns Access is denied error, that means after symlink is created, the smb file share access is not set up immediately, not sure why.
I1208 16:00:27.749206 1664 safe_mounter_windows.go:175] Mount: old name: \var\lib\kubelet\plugins\kubernetes.io\csi\smb.csi.k8s.io\bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\globalmount. new name: c:\var\lib\kubelet\pods\aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa\volumes\kubernetes.io~csi\something-pv-csi\mount
W1208 16:00:27.857327 1664 nodeserver.go:400] ReadDir c:\var\lib\kubelet\pods\aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa\volumes\kubernetes.io~csi\something-pv-csi\mount failed with open c:\var\lib\kubelet\pods\aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaaa\volumes\kubernetes.io~csi\something-pv-csi\mount: Access is denied., unmount this directory
@andyzhangx Thank you for the prompt response.
the two NodePublishVolume operations are for the same pod?
- I can say with certainty, that the operations are indeed the same pod. They are have the same pod ID and pod name.
that means after symlink is created, the smb file share access is not set up immediately, not sure why
- Perhaps the symlink is not ready by the time the second call arrives. I am not certain, but my concern is that the short interval between these operations might not allow enough time for the subsequent call to execute correctly. And that there is a possibility that implementing some form of safety check could prevent this scenario.
@andyzhangx are you planning official release for the below change soon ?
#720 [from andyzhangx/reduce-mount-lock]
@andyzhangx are you planning official release for the below change soon ? #720 [from andyzhangx/reduce-mount-lock]
@vinayaksakharkar I will cut a new release in the middle of this month.
@andyzhangx are you planning official release for the below change soon ? #720 [from andyzhangx/reduce-mount-lock]
@vinayaksakharkar I will cut a new release in the middle of this month.
Thank you @andyzhangx
@tzifudzi
I might have e similar issue to yours. I am seeing access denied while the pod is trying to start. SMB controller logs shows that node publish volume globalmount was successfully mounted but pod fails saying failed to create containerd task failed to creat shim task failed to eval symlinks for mount source \\foldera\golderb\folderc\folderd access denied.
Can you please suggest something or did you ever manage to find a workaround?
@Chil49 We haven't tried driver with changes done by @andyzhangx on master branch. we are waiting for @andyzhangx to cut release in the middle of this month. Meantime, you can mount SMB share on node and use host path for mounting it inside pod as work around if that helps.
@Chil49, the access denied error you're encountering does indicate a potential similarity to the issue I experienced. However, your specific error message differs from mine. While I haven't completely root caused the issue, I suspect that implementing a concurrency lock might help mitigate it. As a temporary workaround, I've found that deleting and then relaunching the pod can resolve the issue. This problem appears to be transient, occurring roughly once in say 500 runs. A retry is usually successful in overcoming this hurdle.
@andyzhangx not sure how you prefer to move forward but I suggest leaving the issue open so people can upvote if they encounter similar errors. Would have tried to help and make the changes but I don't have the free time to take up the task to root cause and resolve.
per #721 (comment), I don't think adding a lock would help since the above logs shows that the first mount succeeded, and then second mount started.
Got it @andyzhangx. I can't provide any further troubleshooting information at this point, nor do I have time to experiment with any code changes that could resolve this. So I will leave it up to you regarding whether the issue should remain open or it can be closed.
@andyzhangx can you please cut release as per your previous comment in the middle of month. We are interested in testing it.
We are facing this issue in our environment. @tzifudzi helped us to raise this issue.
@andyzhangx Are you still planning cut release for code changes you made it for this issue as per your previous comment. You did mention in the middle of this month.
@andyzhangx Are you still planning cut release for code changes you made it for this issue as per your previous comment. You did mention in the middle of this month.
yes, we are waiting for this PR merge: kubernetes/k8s.io#6282, and then new image is ready.
v1.14.0 is already released