Giters

kubernetes-csi / csi-driver-smb

This driver allows Kubernetes to access SMB Server on both Linux and Windows nodes.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Example not working after re-installation

mcreutz opened this issue · comments

commented

What happened

I created a proof-of-concept implementation and installed and un-installed on different clusters. On first installation, the nginx-pod starts successfully. But after un-installation and re-installation, the nginx-pod does not start anymore.

Host 1

AMD64, GCloud VM, Ubuntu, MicroK8s; Initial installation is successful. After re-installation the nginx-pod fails with:

MountVolume.SetUp failed for volume "pv-smb" : rpc error: code = Internal desc = Could not mount 
"/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/8002c928870447b6c7c7b6f56b6e7c310eea90c7a5fc56d1223e4e2977d3bbf3/globalmount" 
at 
"/var/snap/microk8s/common/var/lib/kubelet/pods/eb3916fc-f43e-4251-a887-fbce94c4e5b0/volumes/kubernetes.io~csi/pv-smb/mount": 
mount failed: exit status 32 Mounting command: mount Mounting arguments: -o bind 
/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/8002c928870447b6c7c7b6f56b6e7c310eea90c7a5fc56d1223e4e2977d3bbf3/globalmount 
/var/snap/microk8s/common/var/lib/kubelet/pods/eb3916fc-f43e-4251-a887-fbce94c4e5b0/volumes/kubernetes.io~csi/pv-smb/mount 
Output: mount: 
/var/snap/microk8s/common/var/lib/kubelet/pods/eb3916fc-f43e-4251-a887-fbce94c4e5b0/volumes/kubernetes.io~csi/pv-smb/mount: 
special device 
/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/8002c928870447b6c7c7b6f56b6e7c310eea90c7a5fc56d1223e4e2977d3bbf3/globalmount 
does not exist. dmesg(1) may have more information after failed mount system call.

Interesting is, that the hash value for the plugin (8002c928870447b6c7c7b6f56b6e7c310eea90c7a5fc56d1223e4e2977d3bbf3) does not change with un-install and re-install.

Host 2

AMD64, GCloud VM, Ubuntu, Minikube; Initial installation is successful. After re-installation the nginx-pod fails with:

MountVolume.MountDevice failed for volume "pv-smb" : rpc error: code = Internal desc = volume(smb-server.smb-test.svc.cluster.local/share##) mount "//smb-server.smb-test.svc.cluster.local/share" on "/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/8002c928870447b6c7c7b6f56b6e7c310eea90c7a5fc56d1223e4e2977d3bbf3/globalmount" failed with mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t cifs -o dir_mode=0777,file_mode=0777,vers=3.0,<masked> //smb-server.smb-test.svc.cluster.local/share /var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/8002c928870447b6c7c7b6f56b6e7c310eea90c7a5fc56d1223e4e2977d3bbf3/globalmount
Output: mount error: cifs filesystem not supported by the system
mount error(19): No such device
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log messages (dmesg)

What you expected to happen

After un-installation and re-installation the example pod should start again successfully

How to reproduce it

Install - de-install - install:

Manifests

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: smb
provisioner: smb.csi.k8s.io
parameters:
  source: "//smb-server.smb-test.svc.cluster.local/share"
  # if csi.storage.k8s.io/provisioner-secret is provided, will create a sub directory
  # with PV name under source
  csi.storage.k8s.io/provisioner-secret-name: "smbcreds"
  csi.storage.k8s.io/provisioner-secret-namespace: "smb-test"
  csi.storage.k8s.io/node-stage-secret-name: "smbcreds"
  csi.storage.k8s.io/node-stage-secret-namespace: "smb-test"
volumeBindingMode: Immediate
mountOptions:
  - dir_mode=0777
  - file_mode=0777
  - uid=1001
  - gid=1001
  - noperm
  - mfsymlinks
  - cache=strict
  - noserverino  # required to prevent data corruption

---
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: smb.csi.k8s.io
  name: pv-smb
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: smb
  mountOptions:
    - dir_mode=0777
    - file_mode=0777
    - vers=3.0
  csi:
    driver: smb.csi.k8s.io
    readOnly: false
    # volumeHandle format: {smb-server-address}#{sub-dir-name}#{share-name}
    # make sure this value is unique for every share in the cluster
    volumeHandle: smb-server.smb-test.svc.cluster.local/share##
    volumeAttributes:
      source: "//smb-server.smb-test.svc.cluster.local/share"
    nodeStageSecretRef:
      name: smbcreds
      namespace: smb-test

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-smb
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  storageClassName: smb

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: deployment-smb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
      name: deployment-smb
    spec:
      containers:
        - name: deployment-smb
          image: mcr.microsoft.com/oss/nginx/nginx:1.19.5
          command:
            - "/bin/bash"
            - "-c"
            - set -euo pipefail; while true; do echo $(date) >> /mnt/smb/outfile; sleep 1; done
          volumeMounts:
            - name: smb
              mountPath: "/mnt/smb"
              readOnly: false
      volumes:
        - name: smb
          persistentVolumeClaim:
            claimName: pvc-smb
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate

Installation

#### Namespace
kubectl create ns smb-test

#### Setting up the CSI driver for SMB
curl -skSL https://raw.githubusercontent.com/kubernetes-csi/csi-driver-smb/v1.13.0/deploy/install-driver.sh | bash -s v1.13.0 --

#### Check pod status
kubectl -n kube-system get pod -o wide --watch -l app=csi-smb-controller
kubectl -n kube-system get pod -o wide --watch -l app=csi-smb-node

#### Setting up a SMB test server
kubectl create -n smb-test secret generic smbcreds --from-literal username=USERNAME --from-literal password="PASSWORD" 
kubectl create -n smb-test -f https://raw.githubusercontent.com/kubernetes-csi/csi-driver-smb/master/deploy/example/smb-provisioner/smb-server.yaml 

#### Setting up an example application
kubectl create -f storageclass-smb.yaml
kubectl create -f pv-smb.yaml
kubectl create -f pvc-smb.yaml -n smb-test
kubectl create -f deployment-smb.yaml -n smb-test

Un-installation

#### Delete example application
kubectl delete -f deployment-smb.yaml -n smb-test
kubectl delete -f pvc-smb.yaml -n smb-test
kubectl delete -f pv-smb.yaml
kubectl delete -f storageclass-smb.yaml

#### Delete SMB test server
kubectl delete -n smb-test -f https://raw.githubusercontent.com/kubernetes-csi/csi-driver-smb/master/deploy/example/smb-provisioner/smb-server.yaml 
kubectl delete -n smb-test secret smbcreds 

#### Delete CSI driver for SMB
curl -skSL https://raw.githubusercontent.com/kubernetes-csi/csi-driver-smb/v1.13.0/deploy/uninstall-driver.sh | bash -s --

#### Delete the namespace
kubectl delete ns smb-test

Anything else we need to know?

Environment

kubectl get po -n kube-system -o yaml | grep registry.k8s | grep smb
      image: registry.k8s.io/sig-storage/smbplugin:v1.13.0
      image: registry.k8s.io/sig-storage/smbplugin:v1.13.0
      imageID: docker-pullable://registry.k8s.io/sig-storage/smbplugin@sha256:38b9827870311e5c4da344df8ab6d7856128779233cb4ed5b5a2d1a5a476d86f
      image: registry.k8s.io/sig-storage/smbplugin:v1.13.0
      image: registry.k8s.io/sig-storage/smbplugin:v1.13.0
      imageID: docker-pullable://registry.k8s.io/sig-storage/smbplugin@sha256:38b9827870311e5c4da344df8ab6d7856128779233cb4ed5b5a2d1a5a476d86f
  • CSI Driver version:
  • Kubernetes version (use kubectl version):
kubectl version
Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3
  • OS (e.g. from /etc/os-release): PRETTY_NAME="Ubuntu 22.04.3 LTS"
  • Kernel (e.g. uname -a):
uname -a
Linux ubuntu-desktop 6.2.0-1018-gcp #20~22.04.1-Ubuntu SMP Mon Oct 23 12:29:43 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools: kubectl, see above
  • Others:
commented

there are Microk8s based kubernetes recommended settings, have you configured as following?
https://github.com/kubernetes-csi/csi-driver-smb/tree/master/charts#tips

commented

there are Microk8s based kubernetes recommended settings, have you configured as following? https://github.com/kubernetes-csi/csi-driver-smb/tree/master/charts#tips

Thanks for the prompt response! Will try to install with helm and use the suggested settings. FYI, the 'host 2' I tried to install on is running Minikube. installing 'cifs-utils' from APT on the host did not solve the problem there either.

commented

the error mount error: cifs filesystem not supported by the system is related to Ubuntu kernel version issue: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/2042092, that kernel does not install cifs package, I think you need to install an older version