k3s cluster uninstall / install delete all data in smb volumes
XtraLarge opened this issue · comments
Description
I've a k3s v12.79) cluster with some apps like nextcloud, plex and so on deployed. I've used you csi-smb-driver to mount the data in that pods. Because of working with that data I must enable write access to the volumes.
Yesterday I want to migrate my cluster datatstore from from embedded sqlite to etcd-internal und after that from etcd-internal to etcd external mysql. So I've tried a lot. some node were hard-stopped with k3s-killall.ssh and after that uninstalled with k3s-uninstall.sh. One part of the uninstall script is to the command "rm -rf /var/lib/rancher/k3s". In this step I suddenly saw that an error that a file on my smb-shares could not be removed. I've looked at that shares and all of my data were gone (10 TB of data and my whole life). Fortunately I've a backup of the important data, but the restore will cost 3 days.
What you expected to happen:
The smb-shares should be protected against standard scripts to be deleted. Perhaps it will be a solution to use another location.
How to reproduce it:
Look at the whole description.
Environment:
k8s Environment
k3s v1.27.9
SMB PersistentVolumeClaim
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: 'yes'
creationTimestamp: '2024-01-26T18:05:55Z'
finalizers:
- kubernetes.io/pvc-protection
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:spec:
f:accessModes: {}
f:resources:
f:requests:
.: {}
f:storage: {}
f:storageClassName: {}
f:volumeMode: {}
f:volumeName: {}
manager: agent
operation: Update
time: '2024-01-26T18:05:55Z'
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:pv.kubernetes.io/bind-completed: {}
manager: k3s
operation: Update
time: '2024-01-26T18:10:26Z'
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:status:
f:accessModes: {}
f:capacity:
.: {}
f:storage: {}
f:phase: {}
manager: k3s
operation: Update
subresource: status
time: '2024-01-26T18:10:26Z'
name: pvc-ghost-daten
namespace: xl-nextcloud
resourceVersion: '839'
uid: 2dbdfea8-5005-4a02-8335-b611c6cd6edc
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi
storageClassName: smb
volumeMode: Filesystem
volumeName: pv-ghost-daten
status:
accessModes:
- ReadWriteMany
capacity:
storage: 100Gi
phase: Bound
SMB PersistentVolume:
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-ghost-daten
annotations:
pv.kubernetes.io/bound-by-controller: 'yes'
pv.kubernetes.io/provisioned-by: smb.csi.k8s.io
# key: string
creationTimestamp: '2024-01-26T18:10:19Z'
finalizers:
- kubernetes.io/pv-protection
# - string
labels:
{}
# key: string
resourceVersion: '838'
uid: e112c292-7b18-41b2-87a0-459cf929c158
fields:
- pv-ghost-daten
- 100Gi
- RWX
- Retain
- Bound
- xl-nextcloud/pvc-ghost-daten
- smb
- ''
- 13d
- Filesystem
spec:
accessModes:
- ReadWriteMany
# - string
capacity:
storage: 100Gi
# key: string
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: pvc-ghost-daten
namespace: xl-nextcloud
resourceVersion: '65380748'
uid: 2dbdfea8-5005-4a02-8335-b611c6cd6edc
# fieldPath: string
csi:
controllerExpandSecretRef:
# name: string
# namespace: string
controllerPublishSecretRef:
# name: string
# namespace: string
driver: smb.csi.k8s.io
nodePublishSecretRef:
# name: string
# namespace: string
nodeStageSecretRef:
name: cifs-xtralarge-default-intern
namespace: default
readOnly: false
volumeAttributes:
source: //10.10.10.0/Daten
# key: string
volumeHandle: smb-server.default.svc.cluster.local/ghost-daten
# fsType: string
# nodeExpandSecretRef:
# name: string
# namespace: string
mountOptions:
- dir_mode=0777
- file_mode=0777
# - string
persistentVolumeReclaimPolicy: Retain
storageClassName: smb
volumeMode: Filesystem
# awsElasticBlockStore:
# fsType: string
# partition: int
# readOnly: boolean
# volumeID: string
# azureDisk:
# cachingMode: string
# diskName: string
# diskURI: string
# fsType: string
# kind: string
# readOnly: boolean
# azureFile:
# readOnly: boolean
# secretName: string
# secretNamespace: string
# shareName: string
# cephfs:
# monitors:
# - string
# path: string
# readOnly: boolean
# secretFile: string
# secretRef:
# name: string
# namespace: string
# user: string
# cinder:
# fsType: string
# readOnly: boolean
# secretRef:
# name: string
# namespace: string
# volumeID: string
# fc:
# fsType: string
# lun: int
# readOnly: boolean
# targetWWNs:
# - string
# wwids:
# - string
# flexVolume:
# driver: string
# fsType: string
# options: key: string
# readOnly: boolean
# secretRef:
# name: string
# namespace: string
# flocker:
# datasetName: string
# datasetUUID: string
# gcePersistentDisk:
# fsType: string
# partition: int
# pdName: string
# readOnly: boolean
# glusterfs:
# endpoints: string
# endpointsNamespace: string
# path: string
# readOnly: boolean
# hostPath:
# type: string
# path: string
# iscsi:
# chapAuthDiscovery: boolean
# chapAuthSession: boolean
# fsType: string
# initiatorName: string
# iqn: string
# iscsiInterface: string
# lun: int
# portals:
# - string
# readOnly: boolean
# secretRef:
# name: string
# namespace: string
# targetPortal: string
# local:
# fsType: string
# path: string
# nfs:
# path: string
# readOnly: boolean
# server: string
# nodeAffinity:
# required:
# nodeSelectorTerms:
# - matchExpressions:
# - key: string
# operator: string
# values:
# - string
# matchFields:
# - key: string
# operator: string
# values:
# - string
# photonPersistentDisk:
# fsType: string
# pdID: string
# portworxVolume:
# fsType: string
# readOnly: boolean
# volumeID: string
# quobyte:
# group: string
# readOnly: boolean
# registry: string
# tenant: string
# user: string
# volume: string
# rbd:
# fsType: string
# image: string
# keyring: string
# monitors:
# - string
# pool: string
# readOnly: boolean
# secretRef:
# name: string
# namespace: string
# user: string
# scaleIO:
# fsType: string
# gateway: string
# protectionDomain: string
# readOnly: boolean
# secretRef:
# name: string
# namespace: string
# sslEnabled: boolean
# storageMode: string
# storagePool: string
# system: string
# volumeName: string
# storageos:
# fsType: string
# readOnly: boolean
# secretRef:
# apiVersion: string
# fieldPath: string
# kind: string
# name: string
# namespace: string
# resourceVersion: string
# uid: string
# volumeName: string
# volumeNamespace: string
# vsphereVolume:
# fsType: string
# storagePolicyID: string
# storagePolicyName: string
# volumePath: string
__clone: true](url)
part of the smb volume mount in the nextcloud helm chart:
extraVolumeMounts:
- mountPath: /mnt/GHost_Daten
name: ghost-daten
extraVolumes:
- name: ghost-daten
persistentVolumeClaim:
claimName: pvc-ghost-daten
csi-drivers-smb installed by helm
controller:
affinity: {}
dnsPolicy: ClusterFirstWithHostNet
livenessProbe:
healthPort: 29642
logLevel: 5
metricsPort: 29644
name: csi-smb-controller
nodeSelector: {}
replicas: 1
resources:
csiProvisioner:
limits:
memory: 300Mi
requests:
cpu: 10m
memory: 20Mi
livenessProbe:
limits:
memory: 100Mi
requests:
cpu: 10m
memory: 20Mi
smb:
limits:
memory: 200Mi
requests:
cpu: 10m
memory: 20Mi
runOnControlPlane: false
runOnMaster: false
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
operator: Exists
- effect: NoSchedule
key: node-role.kubernetes.io/controlplane
operator: Exists
- effect: NoSchedule
key: node-role.kubernetes.io/control-plane
operator: Exists
workingMountDir: /tmp
customLabels: {}
driver:
name: smb.csi.k8s.io
feature:
enableGetVolumeStats: true
image:
baseRepo: registry.k8s.io/sig-storage
csiProvisioner:
pullPolicy: IfNotPresent
repository: registry.k8s.io/sig-storage/csi-provisioner
tag: v4.0.0
csiproxy:
pullPolicy: IfNotPresent
repository: ghcr.io/kubernetes-sigs/sig-windows/csi-proxy
tag: v1.1.2
livenessProbe:
pullPolicy: IfNotPresent
repository: registry.k8s.io/sig-storage/livenessprobe
tag: v2.12.0
nodeDriverRegistrar:
pullPolicy: IfNotPresent
repository: registry.k8s.io/sig-storage/csi-node-driver-registrar
tag: v2.10.0
smb:
pullPolicy: IfNotPresent
repository: registry.k8s.io/sig-storage/smbplugin
tag: v1.14.0
linux:
dnsPolicy: ClusterFirstWithHostNet
dsName: csi-smb-node
enabled: true
krb5CacheDirectory: ''
krb5Prefix: ''
kubelet: /var/lib/kubelet
resources:
livenessProbe:
limits:
memory: 100Mi
requests:
cpu: 10m
memory: 20Mi
nodeDriverRegistrar:
limits:
memory: 100Mi
requests:
cpu: 10m
memory: 20Mi
smb:
limits:
memory: 200Mi
requests:
cpu: 10m
memory: 20Mi
tolerations:
- operator: Exists
node:
affinity: {}
livenessProbe:
healthPort: 29643
logLevel: 5
maxUnavailable: 1
nodeSelector: {}
podAnnotations: {}
podLabels: {}
priorityClassName: system-cluster-critical
rbac:
create: true
name: smb
securityContext:
seccompProfile:
type: RuntimeDefault
serviceAccount:
controller: csi-smb-controller-sa
create: true
node: csi-smb-node-sa
windows:
csiproxy:
affinity: {}
dsName: csi-proxy-win
enabled: false
nodeSelector:
kubernetes.io/os: windows
tolerations: {}
username: NT AUTHORITY\SYSTEM
dsName: csi-smb-node-win
enabled: false
kubelet: C:\var\lib\kubelet
removeSMBMappingDuringUnmount: true
resources:
livenessProbe:
limits:
memory: 150Mi
requests:
cpu: 10m
memory: 40Mi
nodeDriverRegistrar:
limits:
memory: 150Mi
requests:
cpu: 10m
memory: 40Mi
smb:
limits:
memory: 200Mi
requests:
cpu: 10m
memory: 40Mi
tolerations:
- effect: NoSchedule
key: node.kubernetes.io/os
operator: Exists
global:
cattle:
clusterId: c-m-j25zxrqn
systemProjectId: p-4vtcx
I found that other volumes like shared local path are also deletet. So it is'nt a csi-smb problem. It is a k3s problem.