AKS pod failed to mount SMB share with error Output: mount error: cifs filesystem not supported by the system mount error(19): No such device

[tommychoy]

What happened:
AKS pod failed to mount SMB share with error.

Warning FailedMount 11s (x46 over 91m) kubelet MountVolume.MountDevice failed for volume "XXX" : rpc error: code = Internal desc = volume(XXX) mount "YYYY" on "/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/5ca65c97f4121533d7420b0fadf19524fdaabf11f7bcf3461b9d4b9dbce66030/globalmount" failed with mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t cifs -o dir_mode=0700,file_mode=0700,uid=1234,gid=1234,vers=2.0, YYYY /var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/5ca65c97f4121533d7420b0fadf19524fdaabf11f7bcf3461b9d4b9dbce66030/globalmount
Output: mount error: cifs filesystem not supported by the system
mount error(19): No such device
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log messages (dmesg)

What you expected to happen:
it shall be able to mount the SMB share without issue using PV and PVC

How to reproduce it:
Create a PV in AKS using the smb.csi.k8s.io (example csi config as below) and create a corresponding PVC. Create a pod to mount the volume.
csi:
driver: smb.csi.k8s.io
nodeStageSecretRef:
name:
namespace:
volumeAttributes:
source:
volumeHandle:
mountOptions:

• dir_mode=0700
• file_mode=0700
• uid=1234
• gid=1234
• vers=2.0

Anything else we need to know?:

Environment:

• CSI Driver version:
  generateName: csi-smb-controller-c57b5db68-
  app: csi-smb-controller
  app.kubernetes.io/name: csi-driver-smb
  helm.sh/chart: csi-driver-smb-v1.12.0
  name: csi-smb-controller-c57b5db68-wwvgg
  name: csi-smb-controller-c57b5db68
  - --drivername=smb.csi.k8s.io
  image: /registry.k8s.io/sig-storage/smbplugin:v1.12.0
  name: smb
  subPath: container_smb.conf
  value: csi-smb-controller-c57b5db68
  value: smb
  value: /registry.k8s.io/sig-storage/smbplugin:v1.12.0
  value: csi-smb-controller
  serviceAccount: csi-smb-controller-sa
  serviceAccountName: csi-smb-controller-sa
  image: /registry.k8s.io/sig-storage/smbplugin:v1.12.0
  imageID: /registry.k8s.io/sig-storage/smbplugin@sha256:c5208e92f6d9bae0f597cab8884a22c8ad7338211abc8a90727120e860e7254d
  name: smb
  generateName: csi-smb-node-
  app: csi-smb-node
  app.kubernetes.io/name: csi-driver-smb
  helm.sh/chart: csi-driver-smb-v1.12.0
  name: csi-smb-node-26pcr
  name: csi-smb-node
  value: /var/lib/kubelet/plugins/smb.csi.k8s.io/csi.sock
  - --drivername=smb.csi.k8s.io
  image: /registry.k8s.io/sig-storage/smbplugin:v1.12.0
  name: smb
  subPath: container_smb.conf
  value: csi-smb-node
  value: smb
  value: /registry.k8s.io/sig-storage/smbplugin:v1.12.0
  value: csi-smb-node
  serviceAccount: csi-smb-node-sa
  serviceAccountName: csi-smb-node-sa
  path: /var/lib/kubelet/plugins/smb.csi.k8s.io
  image: /registry.k8s.io/sig-storage/smbplugin:v1.12.0
  imageID: /registry.k8s.io/sig-storage/smbplugin@sha256:c5208e92f6d9bae0f597cab8884a22c8ad7338211abc8a90727120e860e7254d
  name: smb

• Kubernetes version (use kubectl version):
  Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.6", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

• OS (e.g. from /etc/os-release):
  Ubuntu 22.04.3 LTS

• Kernel (e.g. uname -a):
  6.2.0-1014-azure

• Install tools:

• Others:

[Andy Zhang]

mount error: cifs filesystem not supported by the system means the cifs-utils is not installed on the node, could you install that package, and then try again?

[tommychoy]

Already checked cifs-utils is installed on the node.

Just tested to add one more option (nodfs) to PV and seems it can be mounted. Seems there are some behaviour changes in the cfis module in kernel version 6.2.0-1014-azure. We dont need this nodfs option in kernel version 5.15.0-1049-azure

mountOptions:
dir_mode=0700
file_mode=0700
uid=1234
gid=1234
vers=2.0
nodfs

[Michael Chudinov]

We have got same problem recently in the middle of October 2023.
AKS setup:
Kubernetes version v1.25.11
OS Ubuntu 22.04.2 LTS

#cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
`VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"

Kernel 6.2.0-1016-azure

# uname -a
Linux aks-cbslink-18218187-vmss00000M 6.2.0-1016-azure #16~22.04.1-Ubuntu SMP Tue Oct 10 17:11:51 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

cifs-utils package is installed at the node

# apt -qq list cifs-utils
cifs-utils/jammy-updates,jammy-security,now 2:6.14-1ubuntu0.1 amd64 [installed]

However cifs filesystem is not supported by OS. It is not listed under cat /proc/filesystems

[Andy Zhang]

that's a known bug: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/2042092, it only happens on 6.2.0-1016-azure, you could run following daemonset to install linux-image-6.2.0-1015-azure version on every node, after a few minutes when the kernel is successfully installed, reboot every node manually

kubectl apply -f https://raw.githubusercontent.com/andyzhangx/demo/master/aks/apt-install-1015-kernel-version.yaml

After reboot, run following command to make sure all nodes are on 6.2.0-1015-azure version.

# kubectl get no -o wide
NAME                                STATUS     ROLES   AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
aks-nodepool1-19986461-vmss000002   Ready      agent   23h   v1.27.3   10.224.0.5    <none>        Ubuntu 22.04.3 LTS   6.2.0-1015-azure    containerd://1.7.5-1

[Michael Chudinov]

Manual rollback of all nodes to old version 6.2.0-1015-azure is a bad option.

1. It will not work well with autoscaling when new nodes with 6.2.0-1016-azure will come up at any time.
2. And it obviously is not compatible with AKS auto-updates for Kubernetes. All nodes will be updated to 6.2.0-1016-azure every time a new Kubernetes version is rolled out in Azure.