External cifs mount failed
lratkajec opened this issue · comments
What happened:
After adding PV with smb share to pod, Pod won't start with error from pod events and csi-smb-node pod:
code = Internal desc = Could not mount target "/var/lib/kubelet/pods/5d57b1dc-b645-4f8d-9529-d19045b04560/volumes/kubernetes.io~csi/cifs-fsdoc1/mount": wrong number of fields (expected 6, got 10)
What you expected to happen:
Pods spun up with no issue mounting SMB Volume
How to reproduce it:
Mount any smb share with csi driver and static provisioning of PV-s
Environment:
- CSI Driver version: V 1.11
- Kubernetes version (use
kubectl version): v1.25.8+37a9a08 - OS (e.g. from /etc/os-release): Red Hat Enterprise Linux CoreOS release 4.12
- Kernel (e.g.
uname -a): 4.18.0-372.53.1.el8_6.x86_64 linux - Install tools: kubectl local install script
- Others: Openshift 4.12
could you follow this guide to get mount logs? https://github.com/kubernetes-csi/csi-driver-smb/blob/master/docs/csi-debug.md#case2-volume-mountunmount-failed
Hi, here are the logs:
csi-smb-node.log
cifs-mount.log
csi-smb-node-description.log
I think it's caused by the space in source address, could you try a different address without space?
"source":"//fsdoc/MBU Public/Visa i MasterCard mandati/Mandati
We have sucessfully mounted a share without spacing, but we cant get those with space working, and those are the most important. We tried a combination of escapes, like "\ ", "\40", "\x20" but we are always back to the original error. I think these escapes are the right ones, because when you miss an escape, you get a different error and also the logs show the correct path and a success message, for example:
I0612 12:38:27.519793 1 utils.go:77] GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/cbdd02101449c2b6fb35b266ba54261abec6282fc051ae8d58d1a0f9cfee5985/globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["dir_mode=0777","file_mode=0777","uid=4017","gid=4017"],"volume_mount_group":"1001150000"}},"access_mode":{"mode":5}},"volume_context":{"source":"//fsdoc/MBU Public/Visa i MasterCard mandati/Mandati"},"volume_id":"cifs-mandati"}
1098I0612 12:38:27.520021 1 nodeserver.go:206] NodeStageVolume: targetPath(/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/cbdd02101449c2b6fb35b266ba54261abec6282fc051ae8d58d1a0f9cfee5985/globalmount) volumeID(cifs-mandati) context(map[source://fsdoc/MBU Public/Visa i MasterCard mandati/Mandati]) mountflags([dir_mode=0777 file_mode=0777 uid=4017 gid=4017]) mountOptions([dir_mode=0777 file_mode=0777 uid=4017 gid=4017])
1099I0612 12:38:27.522702 1 mount_linux.go:220] Mounting cmd (mount) with arguments (-t cifs -o dir_mode=0777,file_mode=0777,uid=4017,gid=4017,<masked> //fsdoc/MBU Public/Visa i MasterCard mandati/Mandati /var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/cbdd02101449c2b6fb35b266ba54261abec6282fc051ae8d58d1a0f9cfee5985/globalmount)
1100I0612 12:38:27.540730 1 nodeserver.go:238] volume(cifs-mandati) mount "//fsdoc/MBU Public/Visa i MasterCard mandati/Mandati" on "/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/cbdd02101449c2b6fb35b266ba54261abec6282fc051ae8d58d1a0f9cfee5985/globalmount" succeeded
it failed here with the mount info: https://github.com/kubernetes/kubernetes/blob/18d05b646d09b2971dc5400bc288062b0414e8cf/staging/src/k8s.io/mount-utils/mount_helper_unix.go#L109
utils.go:81] GRPC error: rpc error: code = Internal desc = Could not mount target "/var/lib/kubelet/pods/cf4452c3-b24b-4711-84c1-55e8a489af18/volumes/kubernetes.io~csi/cifs-fsdoc1/mount": wrong number of fields (expected 6, got 10)
that could be still due to the space issue.
So there is no workaround if there are whitespace characters in the mount path name? Because this line of code will always split on the whitespaces and we will end up with "wrong number of fields" error: https://github.com/kubernetes/kubernetes/blob/18d05b646d09b2971dc5400bc288062b0414e8cf/staging/src/k8s.io/mount-utils/mount_helper_unix.go#L107
Hi, has this been dealt with?
I'm concerned as this is the codebase/version that will be used for OpenShift 4.12 installs, which causes all cifs mounts with spaces or spaces and escapes to fail.
Is anyone willing to produce a newer release if a PR with the fix was put in?
To get around this, you can use gcr.io/k8s-staging-sig-storage/smbplugin:canary instead of registry.k8s.io/sig-storage/smbplugin:v1.12.0 as it actually works with spaces too.
Thank you!
@zwurv gcr.io/k8s-staging-sig-storage/smbplugin:canary is not working for me with spaces. Did you try anything else along with this?
@venesh0709 did you try any of the other images in there?
gcr.io/k8s-staging-sig-storage/smbplugin@sha256:87c6b8277db6fd9e2910052c679bdb045f91174ff9e537a942048b7f29964bb5
it's quite difficult to know which codebase canary was based on. I wish I could find the code that produces this canary build as I do not believe it's main branch on this project. Canary is version v1.13.0, releases is v1.12.0
What version of Kubernetes are you running?
ocpv4.12.25
K8s v1.25.11
@zwurv
@venesh0709
We upgraded to OCP 4.12 from 4.10 and ended up downgrading back to 4.10 afterwards because of this specific problem.
We haven't figured out how to fix this yet and need to upgrade again to 4.12 soon, sounds like this canary build in staging sig storage won't work, I thought it was that.
What are you running this on? ESX/i (what version?) or?
Sounds like it's really a change in Kubernetes 1.25 from 1.23 (OCP 4.10).
Please let me know if you figure this out, we're struggling with this too.
@zwurv Hey, May I know how did you downgrade the OCP cluster to previous version?
We are running on AWS.
Did you try the escape character?
@venesh0709 Hi, we always use Ansible for the creation/destruction of the cluster, in our case we used that script, destroyed the cluster and rebuilt, and finally used build & release pipelines to redeploy our apps. There's no downgrading of OpenShift.
We tried the escape character and that didn't work.
I'm uncertain if the issue is just this character (which we never used prior to this and use this staging v1.13 docker image).
I'm wondering if this problem is mixed with PodSecurityPolicy deprecation.
https://connect.redhat.com/en/blog/important-openshift-changes-pod-security-standards
@zwurv It seems like all the roads are blocked. RedHat do not support CIFS. So, they might have taken out something from 4.12(Removed APIs) which directly affected CIFS with spaces functionality. Red Hat would not even knew about CIFS failure as they never would have tested CIFS. What are your plans as of now?
Hi @venesh0709, I believe support was removed as of OpenShift 4.11. One of my team members is working on a resolution and once he finds it I'll ask his permission to post here so you can be informed. This will likely be solved within the next week, although I don't know what sort of resolution we'll have.
hi @venesh0709, looks like a colleague of mine found a way around this.
If you've got cifs mounts with paths with spaces in them, create a symlink without spaces on the windows server that points to the path with the spaces, and use that new symlink path without spaces in your pv. This seems to work, we gave it a quick test.
Let us know if this works :)
@zwurv We have tried that and mounting was successful but unfortunately we did not same files in symlink(without spaces) and path with spaces. However, we are glad that it is working for you.