File access/permission issue when trying to install in MicroK8s
struffel opened this issue · comments
What happened:
I tried installing v1.11 of the CSI driver in Microk8s (on ubuntu 22.04) using helm, making sure to add the extra parameter for the microk8s kubelet path.
The installation went well in the sense that all pods (controller+nodes) are running and the storageclass has been created.
However, when attempting to deploy the sample deployment the container ultimately gets stuck in the ContainerCreating stage, citing file access issues:
MountVolume.MountDevice failed for volume "pvc-e2e4fee0-fa4a-4425-a429-2f73e0e6a3e1" : rpc error: code = Internal desc = MkdirAll /var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/ba511973ea796cab301513c3d994a26ea21b6ba12cb3ded452208b3b703c0d4a/globalmount failed with error: mkdir /var/snap: read-only file system
microk8s kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
...
csi-smb-node-2rm6b 3/3 Running 0 2d17h
csi-smb-controller-6b98c5d766-7hr95 3/3 Running 0 2d17h
csi-smb-node-lfnxh 3/3 Running 0 2d17h
csi-smb-node-mkvbq 3/3 Running 0 2d17h
microk8s kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
...
deployment-smb 0/1 1 0 2d17h
microk8s kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc-smb Bound pvc-e2e4fee0-fa4a-4425-a429-2f73e0e6a3e1 1Gi RWX lwk-dfs-cestorage-smb 2d17h
microk8s kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-e2e4fee0-fa4a-4425-a429-2f73e0e6a3e1 1Gi RWX Delete Bound default/pvc-smb lwk-dfs-cestorage-smb 2d17h
# The storage server has been cencored from the output
apiVersion: v1
items:
- apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"lwk-dfs-cestorage-smb"},"mountOptions":["dir_mode=0777","file_mode=0777","uid=1001","gid=1001","noperm","mfsymlinks","cache=strict","noserverino"],"parameters":{"csi.storage.k8s.io/node-stage-secret-name":"lwk-dfs-credentials","csi.storage.k8s.io/node-stage-secret-namespace":"kube-system","csi.storage.k8s.io/provisioner-secret-name":"lwk-dfs-credentials","csi.storage.k8s.io/provisioner-secret-namespace":"kube-system","source":"//x/y"},"provisioner":"smb.csi.k8s.io","reclaimPolicy":"Delete","volumeBindingMode":"Immediate"}
creationTimestamp: "2023-07-28T11:49:15Z"
name: lwk-dfs-cestorage-smb
resourceVersion: "292392"
uid: 66e6f7b5-5e2a-4677-bbcb-f16664c981d4
mountOptions:
- dir_mode=0777
- file_mode=0777
- uid=1001
- gid=1001
- noperm
- mfsymlinks
- cache=strict
- noserverino
parameters:
csi.storage.k8s.io/node-stage-secret-name: lwk-dfs-credentials
csi.storage.k8s.io/node-stage-secret-namespace: kube-system
csi.storage.k8s.io/provisioner-secret-name: lwk-dfs-credentials
csi.storage.k8s.io/provisioner-secret-namespace: kube-system
source: //x/y
provisioner: smb.csi.k8s.io
reclaimPolicy: Delete
volumeBindingMode: Immediate
kind: List
metadata:
resourceVersion: ""
Here is the pod that does not start properly:
microk8s kubectl describe pod deployment-smb-7969c4d48d-t7m7g
Name: deployment-smb-7969c4d48d-t7m7g
Namespace: default
Priority: 0
Service Account: default
Node: lwk-ec-worker2/130.3.141.132
Start Time: Mon, 31 Jul 2023 05:17:27 +0000
Labels: app=nginx
pod-template-hash=7969c4d48d
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/deployment-smb-7969c4d48d
Containers:
deployment-smb:
Container ID:
Image: mcr.microsoft.com/oss/nginx/nginx:1.19.5
Image ID:
Port: <none>
Host Port: <none>
Command:
/bin/bash
-c
set -euo pipefail; while true; do echo $(date) >> /mnt/smb/outfile; sleep 1; done
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/mnt/smb from smb (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vldng (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
smb:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: pvc-smb
ReadOnly: false
kube-api-access-vldng:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 3m18s (x19 over 25m) kubelet MountVolume.MountDevice failed for volume "pvc-e2e4fee0-fa4a-4425-a429-2f73e0e6a3e1" : rpc error: code = Internal desc = MkdirAll /var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/ba511973ea796cab301513c3d994a26ea21b6ba12cb3ded452208b3b703c0d4a/globalmount failed with error: mkdir /var/snap: read-only file system
Warning FailedMount 3m17s (x10 over 23m) kubelet Unable to attach or mount volumes: unmounted volumes=[smb], unattached volumes=[], failed to process volumes=[]: timed out waiting for the condition
In the last code window the error is apparent, the driver can not mount the volume on the worker node, because it lacks permissions (?).
What you expected to happen:
Given the explicit instructions for MicroK8s in the guide I expected the final deployment pod to not run into file access/permission (?) issues when trying to use the driver with Microk8s.
How to reproduce it:
Follow the guide for installing v1.11 of the driver, including the special kubelet instruction for microk8s.
Create the storage class.
Try using the storage class in the demo deployment.
Anything else we need to know?:
Environment:
- CSI Driver version:
# The given example command generated no output, so here is the yaml for the controller pod:
microk8s kubectl get po -n kube-system csi-smb-controller-6b98c5d766-7hr95 -o yaml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2023-07-28T11:47:32Z"
generateName: csi-smb-controller-6b98c5d766-
labels:
app: csi-smb-controller
app.kubernetes.io/instance: csi-driver-smb
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: csi-driver-smb
app.kubernetes.io/version: v1.11.0
helm.sh/chart: csi-driver-smb-v1.11.0
pod-template-hash: 6b98c5d766
name: csi-smb-controller-6b98c5d766-7hr95
namespace: kube-system
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: csi-smb-controller-6b98c5d766
uid: bc1dec85-21d8-4c9f-b803-343f3ef606bc
resourceVersion: "290687"
uid: 61f9dbaa-c71d-4feb-a364-c087b101cf85
spec:
containers:
- args:
- -v=2
- --csi-address=$(ADDRESS)
- --leader-election
- --leader-election-namespace=kube-system
- --extra-create-metadata=true
env:
- name: ADDRESS
value: /csi/csi.sock
image: registry.k8s.io/sig-storage/csi-provisioner:v3.5.0
imagePullPolicy: IfNotPresent
name: csi-provisioner
resources:
limits:
memory: 300Mi
requests:
cpu: 10m
memory: 20Mi
securityContext:
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /csi
name: socket-dir
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-lbrrk
readOnly: true
- args:
- --csi-address=/csi/csi.sock
- --probe-timeout=3s
- --health-port=29642
- --v=2
image: registry.k8s.io/sig-storage/livenessprobe:v2.10.0
imagePullPolicy: IfNotPresent
name: liveness-probe
resources:
limits:
memory: 100Mi
requests:
cpu: 10m
memory: 20Mi
securityContext:
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /csi
name: socket-dir
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-lbrrk
readOnly: true
- args:
- --v=5
- --endpoint=$(CSI_ENDPOINT)
- --metrics-address=0.0.0.0:29644
- --drivername=smb.csi.k8s.io
- --working-mount-dir=/tmp
env:
- name: CSI_ENDPOINT
value: unix:///csi/csi.sock
image: registry.k8s.io/sig-storage/smbplugin:v1.11.0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 5
httpGet:
path: /healthz
port: healthz
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 10
name: smb
ports:
- containerPort: 29642
hostPort: 29642
name: healthz
protocol: TCP
- containerPort: 29644
hostPort: 29644
name: metrics
protocol: TCP
resources:
limits:
memory: 200Mi
requests:
cpu: 10m
memory: 20Mi
securityContext:
privileged: true
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /csi
name: socket-dir
- mountPath: /tmp
name: tmp-dir
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-lbrrk
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
enableServiceLinks: true
hostNetwork: true
nodeName: lwk-ec-controller
nodeSelector:
kubernetes.io/os: linux
node-role.kubernetes.io/control-plane: ""
preemptionPolicy: PreemptLowerPriority
priority: 2000000000
priorityClassName: system-cluster-critical
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
seccompProfile:
type: RuntimeDefault
serviceAccount: csi-smb-controller-sa
serviceAccountName: csi-smb-controller-sa
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
operator: Exists
- effect: NoSchedule
key: node-role.kubernetes.io/controlplane
operator: Exists
- effect: NoSchedule
key: node-role.kubernetes.io/control-plane
operator: Exists
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- emptyDir: {}
name: socket-dir
- emptyDir: {}
name: tmp-dir
- name: kube-api-access-lbrrk
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
...
- Kubernetes version (use
kubectl version):
kubectl version
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.2", GitCommit:"7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647", GitTreeState:"clean", BuildDate:"2023-05-28T05:41:33Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.2", GitCommit:"7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647", GitTreeState:"clean", BuildDate:"2023-05-28T05:42:25Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"linux/amd64"}
- OS (e.g. from /etc/os-release):
cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
- Kernel (e.g.
uname -a):
uname -a
Linux lwk-ec-controller 5.15.0-78-generic #85-Ubuntu SMP Fri Jul 7 15:25:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
- Install tools:
snap, as is usual for microk8s.
snap --version
snap 2.59.5
snapd 2.59.5
series 16
ubuntu 22.04
kernel 5.15.0-78-generic
- Others:
Update: I downgraded to v1.10 and the error (kinda) disappeared. The pods can now start. However, the outfile still does not show up on the SMB share, it remains only in the container.
Update: When running df in a container with a mounted volume we can see that it maps to the host's hard disk rather thatn the SMB share:
microk8s kubectl exec deployment-smb-7969c4d48d-zv85t -- df -h
Filesystem Size Used Avail Use% Mounted on
overlay 98G 11G 83G 11% /
tmpfs 64M 0 64M 0% /dev
/dev/mapper/ubuntu--vg-ubuntu--lv 98G 11G 83G 11% /mnt/smb
shm 64M 0 64M 0% /dev/shm
tmpfs 7.7G 12K 7.7G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 3.9G 0 3.9G 0% /proc/acpi
tmpfs 3.9G 0 3.9G 0% /proc/scsi
tmpfs 3.9G 0 3.9G 0% /sys/firmware
So the volume is provisioned on the SMB share, the pvc gets fulfilled, the pod gets started but the actual mount does not exist.
Update: I simply misspelt the 'linux.kubelet' parameter, meaning that the custom kubelet path for microk8s wasn't applied.