No directory deletion when using guest access

Question

No directory deletion when using guest access

sprat opened this issue 9 months ago · comments

What happened:

The directories created to support the PVCs are not deleted by the controller when using the guest mount option to access the SMB share. I think it is expected to work since there is some code that handles this special case.

What you expected to happen:

I would expect the PVC directory to be deleted, as there is some code to support that.

How to reproduce it:

I installed the driver using the helm charts and used the latest version available (v1.12.0).

Here is the storage class manifest I used:

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: smb-csi
provisioner: smb.csi.k8s.io
parameters:
  source: //smb-server.default.svc.cluster.local/public
reclaimPolicy: Delete
mountOptions:
  - guest
  - noperm  # don't check permissions at client side
  - mfsymlinks  # allow symlinks
  - cache=strict  # use a cache strategy which assures coherency when multiple clients access the same data
  - noserverino  # required to prevent data corruption

And the PVC manifest:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc
spec:
  storageClassName: smb-csi
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 50G

Finally, here are the logs of the controller pod:

[root@masterworker1 ~]# kubectl -n kube-system logs csi-smb-controller-56bbbd5766-g9cvr smb
W1027 12:32:54.212503       1 main.go:61] nodeid is empty
I1027 12:32:54.212742       1 main.go:94] set up prometheus server on [::]:29644
I1027 12:32:54.212887       1 smb.go:93]
DRIVER INFORMATION:
-------------------
Build Date: "2023-08-11T13:37:11Z"
Compiler: gc
Driver Name: smb.csi.k8s.io
Driver Version: v1.12.0
Git Commit: ""
Go Version: go1.19
Platform: linux/amd64

Streaming logs below:
I1027 12:32:54.215108       1 mount_linux.go:284] Detected umount with safe 'not mounted' behavior
I1027 12:32:54.215162       1 driver.go:93] Enabling controller service capability: CREATE_DELETE_VOLUME
I1027 12:32:54.215169       1 driver.go:93] Enabling controller service capability: SINGLE_NODE_MULTI_WRITER
I1027 12:32:54.215173       1 driver.go:93] Enabling controller service capability: CLONE_VOLUME
I1027 12:32:54.215179       1 driver.go:112] Enabling volume access mode: SINGLE_NODE_WRITER
I1027 12:32:54.215183       1 driver.go:112] Enabling volume access mode: SINGLE_NODE_READER_ONLY
I1027 12:32:54.215186       1 driver.go:112] Enabling volume access mode: SINGLE_NODE_SINGLE_WRITER
I1027 12:32:54.215190       1 driver.go:112] Enabling volume access mode: SINGLE_NODE_MULTI_WRITER
I1027 12:32:54.215194       1 driver.go:112] Enabling volume access mode: MULTI_NODE_READER_ONLY
I1027 12:32:54.215198       1 driver.go:112] Enabling volume access mode: MULTI_NODE_SINGLE_WRITER
I1027 12:32:54.215203       1 driver.go:112] Enabling volume access mode: MULTI_NODE_MULTI_WRITER
I1027 12:32:54.215207       1 driver.go:103] Enabling node service capability: STAGE_UNSTAGE_VOLUME
I1027 12:32:54.215211       1 driver.go:103] Enabling node service capability: SINGLE_NODE_MULTI_WRITER
I1027 12:32:54.215217       1 driver.go:103] Enabling node service capability: VOLUME_MOUNT_GROUP
I1027 12:32:54.215221       1 driver.go:103] Enabling node service capability: GET_VOLUME_STATS
I1027 12:32:54.215360       1 server.go:118] Listening for connections on address: &net.UnixAddr{Name:"//csi/csi.sock", Net:"unix"}
I1027 12:32:54.866717       1 utils.go:76] GRPC call: /csi.v1.Identity/GetPluginInfo
I1027 12:32:54.866732       1 utils.go:77] GRPC request: {}
I1027 12:32:54.867847       1 utils.go:83] GRPC response: {"name":"smb.csi.k8s.io","vendor_version":"v1.12.0"}
I1027 12:32:54.868250       1 utils.go:76] GRPC call: /csi.v1.Identity/GetPluginCapabilities
I1027 12:32:54.868266       1 utils.go:77] GRPC request: {}
I1027 12:32:54.868302       1 utils.go:83] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}}]}
I1027 12:32:54.868885       1 utils.go:76] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I1027 12:32:54.868894       1 utils.go:77] GRPC request: {}
I1027 12:32:54.868920       1 utils.go:83] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":13}}},{"Type":{"Rpc":{"type":7}}}]}
I1027 12:32:55.071194       1 utils.go:76] GRPC call: /csi.v1.Identity/GetPluginInfo
I1027 12:32:55.071208       1 utils.go:77] GRPC request: {}
I1027 12:32:55.071246       1 utils.go:83] GRPC response: {"name":"smb.csi.k8s.io","vendor_version":"v1.12.0"}
I1027 12:58:49.934020       1 utils.go:76] GRPC call: /csi.v1.Controller/CreateVolume
I1027 12:58:49.934042       1 utils.go:77] GRPC request: {"capacity_range":{"required_bytes":50000000000},"name":"pvc-a0f7c2ca-699d-4528-a148-724e46985ee5","parameters":{"csi.storage.k8s.io/pv/name":"pvc-a0f7c2ca-699d-4528-a148-724e46985ee5","csi.storage.k8s.io/pvc/name":"test-pvc","csi.storage.k8s.io/pvc/namespace":"default","source":"//smb-server.default.svc.cluster.local/public"},"volume_capabilities":[{"AccessType":{"Mount":{"mount_flags":["guest","noperm","mfsymlinks","cache=strict","noserverino"]}},"access_mode":{"mode":5}}]}
I1027 12:58:49.934293       1 controllerserver.go:93] guest mount option([guest noperm mfsymlinks cache=strict noserverino]) is provided, create subdirectory
I1027 12:58:49.934305       1 controllerserver.go:269] internally mounting //smb-server.default.svc.cluster.local/public at /tmp/pvc-a0f7c2ca-699d-4528-a148-724e46985ee5
I1027 12:58:49.934403       1 nodeserver.go:206] NodeStageVolume: targetPath(/tmp/pvc-a0f7c2ca-699d-4528-a148-724e46985ee5) volumeID(smb-server.default.svc.cluster.local/public#pvc-a0f7c2ca-699d-4528-a148-724e46985ee5#) context(map[source://smb-server.default.svc.cluster.local/public]) mountflags([guest noperm mfsymlinks cache=strict noserverino]) mountOptions([guest noperm mfsymlinks cache=strict noserverino])
I1027 12:58:49.934885       1 mount_linux.go:245] Detected OS without systemd
I1027 12:58:49.934896       1 mount_linux.go:220] Mounting cmd (mount) with arguments (-t cifs -o guest,noperm,mfsymlinks,cache=strict,noserverino //smb-server.default.svc.cluster.local/public /tmp/pvc-a0f7c2ca-699d-4528-a148-724e46985ee5)
I1027 12:58:50.067581       1 nodeserver.go:238] volume(smb-server.default.svc.cluster.local/public#pvc-a0f7c2ca-699d-4528-a148-724e46985ee5#) mount "//smb-server.default.svc.cluster.local/public" on "/tmp/pvc-a0f7c2ca-699d-4528-a148-724e46985ee5" succeeded
I1027 12:58:50.069367       1 controllerserver.go:287] internally unmounting /tmp/pvc-a0f7c2ca-699d-4528-a148-724e46985ee5
I1027 12:58:50.069390       1 nodeserver.go:260] NodeUnstageVolume: CleanupMountPoint on /tmp/pvc-a0f7c2ca-699d-4528-a148-724e46985ee5 with volume smb-server.default.svc.cluster.local/public#pvc-a0f7c2ca-699d-4528-a148-724e46985ee5#
I1027 12:58:50.069547       1 mount_helper_common.go:93] unmounting "/tmp/pvc-a0f7c2ca-699d-4528-a148-724e46985ee5" (corruptedMount: false, mounterCanSkipMountPointChecks: true)
I1027 12:58:50.069559       1 mount_linux.go:362] Unmounting /tmp/pvc-a0f7c2ca-699d-4528-a148-724e46985ee5
I1027 12:58:50.096179       1 mount_helper_common.go:150] Warning: deleting path "/tmp/pvc-a0f7c2ca-699d-4528-a148-724e46985ee5"
I1027 12:58:50.096272       1 nodeserver.go:269] NodeUnstageVolume: unmount volume smb-server.default.svc.cluster.local/public#pvc-a0f7c2ca-699d-4528-a148-724e46985ee5# on /tmp/pvc-a0f7c2ca-699d-4528-a148-724e46985ee5 successfully
I1027 12:58:50.096286       1 utils.go:83] GRPC response: {"volume":{"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-a0f7c2ca-699d-4528-a148-724e46985ee5","csi.storage.k8s.io/pvc/name":"test-pvc","csi.storage.k8s.io/pvc/namespace":"default","source":"//smb-server.default.svc.cluster.local/public","subdir":"pvc-a0f7c2ca-699d-4528-a148-724e46985ee5"},"volume_id":"smb-server.default.svc.cluster.local/public#pvc-a0f7c2ca-699d-4528-a148-724e46985ee5#"}}
I1027 12:59:34.412880       1 utils.go:76] GRPC call: /csi.v1.Controller/DeleteVolume
I1027 12:59:34.412915       1 utils.go:77] GRPC request: {"volume_id":"smb-server.default.svc.cluster.local/public#pvc-a0f7c2ca-699d-4528-a148-724e46985ee5#"}
I1027 12:59:34.413330       1 controllerserver.go:188] DeleteVolume(smb-server.default.svc.cluster.local/public#pvc-a0f7c2ca-699d-4528-a148-724e46985ee5#) does not delete subdirectory
I1027 12:59:34.413355       1 utils.go:83] GRPC response: {}

As you can see in the logs, during the CreateVolume, the guest mount option is properly detected, but not during the DeleteVolume (does not delete subdirectory wording)... but the mount options did not change in-between.

Environment:

Yes, I know, it's not an up-to-date environment, but I don't think it's related to the issue here.

[root@masterworker1 ~]# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:22:29Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:15:38Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}

[root@masterworker1 ~]# kubectl get po -n kube-system -o yaml | grep smb | grep reg
      image: registry.k8s.io/sig-storage/smbplugin:v1.12.0
      image: registry.k8s.io/sig-storage/smbplugin:v1.12.0
      imageID: registry.k8s.io/sig-storage/smbplugin@sha256:885c1654bdffc8c034ba65798728a40022d6d2791ecf23c8b844955c22a27c79
      image: registry.k8s.io/sig-storage/smbplugin:v1.12.0
      image: registry.k8s.io/sig-storage/smbplugin:v1.12.0
      imageID: registry.k8s.io/sig-storage/smbplugin@sha256:885c1654bdffc8c034ba65798728a40022d6d2791ecf23c8b844955c22a27c79
      image: registry.k8s.io/sig-storage/smbplugin:v1.12.0
      image: registry.k8s.io/sig-storage/smbplugin:v1.12.0
      imageID: registry.k8s.io/sig-storage/smbplugin@sha256:885c1654bdffc8c034ba65798728a40022d6d2791ecf23c8b844955c22a27c79
      image: registry.k8s.io/sig-storage/smbplugin:v1.12.0
      image: registry.k8s.io/sig-storage/smbplugin:v1.12.0
      imageID: registry.k8s.io/sig-storage/smbplugin@sha256:885c1654bdffc8c034ba65798728a40022d6d2791ecf23c8b844955c22a27c79

[root@masterworker1 ~]# cat /etc/os-release
NAME="AlmaLinux"
VERSION="8.5 (Arctic Sphynx)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.5"
PLATFORM_ID="platform:el8"
PRETTY_NAME="AlmaLinux 8.5 (Arctic Sphynx)"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:almalinux:almalinux:8::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"

ALMALINUX_MANTISBT_PROJECT="AlmaLinux-8"
ALMALINUX_MANTISBT_PROJECT_VERSION="8.5"

[root@masterworker1 ~]# uname -a
Linux masterworker1 4.18.0-348.el8.x86_64 #1 SMP Tue Nov 9 06:28:28 EST 2021 x86_64 x86_64 x86_64 GNU/Linux

Sylvain Prat · Answer 1 · Fri Oct 27 2023 21:40:12 GMT+0800 (China Standard Time)

Note: I also tried to put the guest option in a mountOptions field inside a secret referenced in the storage class parameters, but it does not work either, it's even worse since CreateVolume fails.

Sylvain Prat · Answer 2 · Fri Oct 27 2023 22:26:59 GMT+0800 (China Standard Time)

After some iterations, I finally found one way to make it work:

create a secret containing mountOptions: guest
use this secret for csi.storage.k8s.io/provisioner-secret-name but not for csi.storage.k8s.io/node-stage-secret-name (otherwise it fails because there's no username defined)
add guest in the mountOptions of the storage class

I had a couple of surprises due to the asymetry between CreateVolume and DeleteVolume:

CreateVolume ignores the mountOptions defined in the secret
DeleteVolume ignores the mountOptions defined in the storage class

So we need to add the guest option in 2 different places, in a secret and in the storage class. It's hard to understand by reading the documentation, maybe this should be clarified in the documentation. Or maybe the implementation could be improved to simplify the usage, e.g. by merging the mountOptions defined in the secret with the mountOptions defined in the storage class?

Andy Zhang · Answer 3 · Sat Nov 04 2023 23:38:23 GMT+0800 (China Standard Time)

that's because DeleteVolume does not take mountOptions defined in the storage class, as long as you provide csi.storage.k8s.io/provisioner-secret-name, it would delete the volume.

Here is the doc link of this issue: https://github.com/kubernetes-csi/csi-driver-smb/blob/master/docs/driver-parameters.md#provide-mountoptions-for-deletevolume

Sylvain Prat · Answer 4 · Mon Nov 06 2023 17:01:03 GMT+0800 (China Standard Time)

Yes, I have finally understood how it works, but it was after some trials and errors! That's why I suggest to either better document or simplify the process (if possible, I don't know).