Unable to mount volumes for pod

[Emmanuel COLUSSI]

Hello ,

I'm using the hpe plugin to create dynamically volumes on a Nimble storage for deployment of SQL server 2019 and and the docker volume does not mount.
My configuration is as follows :
3 physical servers
1 master node + 2 nodes
Red Hat Enterprise Linux 7.6
Kubernetes release 1.15
Docker 18.09.7
Linux Nimble storage toolkit 2.5.1.110
1 Nimble Storage os release 5.0.7.200-612527
1 dedicated iSCSI VLAN

My problem is :
Refresh iSCSI is not automatic and the docker volume is not mounted because data access is not configured on nimble storage and the SQL Server pods,can not be deployed because it is impossible to mount the volume.

when I look at the log of the deployment of the SQL container I have the following :

· Warning FailedMount 3m28s (x142 over 5h22m) kubelet, dl35 Unable to mount volumes for pod "mssql-kjflr_default(0d48ba77-4490-4b24-b74d-779a4e350f97)": timeout expired waiting for volumes to attach or mount for pod "default"/"mssql-kjflr". list of unmounted volumes=[mssql]. list of unattached volumes=[mssql default-token-nw7fd]
· Warning FailedMount (x128 over 5h24m) kubelet, dl35 MountVolume.SetUp failed for volume "database-ef0b7359-a206-4aae-9720-618985e6c124" : mount command failed, status: Failure, reason: Post http://unix/VolumeDriver.Mount: http: ContentLength=101 with Body length 0

what works :
volume creation on the Nimble storage
the docker volume is created : the PVC and PV are ok

The procedure I followed is as follows:

· I manually run the doryd process to trace the logs :
/usr/libexec/kubernetes/kubelet-plugins/volume/exec/hpe.com~nimble/doryd /root/.kube/config hpe.com

· creating a storage class :
· creating a PVC
· creating a SQL Server pods

I hope my explanations are clear ...

I do not know if you have an idea or look for the problem ? did I forget a step?

thank you in advance for your help.

Thanks

Emmanuel

Config Files :

/usr/libexec/kubernetes/kubelet-plugins/volume/exec/hpe.com~nimble/nimble.json
{
"dockerVolumePluginSocketPath": "/run/docker/plugins/nimble.sock",
"logFilePath": "/var/log/dory.log",
"logDebug": false,
"stripK8sFromOptions": true,
"createVolumes": true,
"enable1.6": false,
"listOfStorageResourceOptions": [ "size", "sizeInGiB" ],
"factorForConversion": 1073741824,
"defaultOptions":[ {"mountConflictDelay": 30}]
}

Storage class :

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: database
provisioner: hpe.com/nimble
parameters:
description: "Volume provisioned by doryd : Nimble SQLServer DATA KUB"
perfPolicy: "SQL Server"
folder: "kub-SQLDATA"

Create PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: kb-sqlserver-data01
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: database

Deployment SQL POD

apiVersion: extensions/v1beta1
kind: ReplicaSet
metadata:
name: mssql
labels:
app: mssql
spec:
replicas: 1
selector:
matchLabels:
app: mssql
template:
metadata:
labels:
app: mssql
spec:
containers:
- name: mssql
image: mcr.microsoft.com/mssql/rhel/server:2019-CTP3.1
resources:
requests:
cpu: 4
memory: 16Gi
env:
- name: ACCEPT_EULA
value: "Y"
- name: SA_PASSWORD
value: "********"
ports:

• containerPort: 1433
  volumeMounts:
  - name: mssql
  mountPath: /var/opt/mssql
  volumes:
  - name: mssql
  persistentVolumeClaim:
  claimName: kb-sqlserver-data01