Rook block device not working for ReadWriteMany

[Henry Whitaker]

Whe trying to create a PersistentVolumeClaim with ReadWriteMany, I get the following error when running kubectl describe pvc <name>:

rook multi node access modes are only supported on rbd `block` type volumes

My cluster:yml:

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook
  namespace: rook
spec:
  backend: ceph
  hostNetwork: false
  dataDirHostPath: /storage
  placement:
    all:
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Equal
        effect: NoSchedule
  cephVersion:
    image: ceph/ceph:v14.2.4-20190917
  mon:
    count: 3
    allowMultiplePerNode: false
  storage:
    useAllNodes: true
    useAllDevices: false
    storeConfig:
      databaseSizeMB: 1024
      journalSizeMB: 1024
    # Uncomment the following line and replace it with the name of block device used for storage:
    deviceFilter: sdb
    # Uncomment the following lines when using a directory for storage:
    #directories:
    #- path: /storage/data

/dev/sdb it works for ReadWriteOnce claims though, any ideas?

(Also, can create a separate issue if needed, but might be worth updating the image used for the ingress controller as the nginx version in the default is fairly old, I used k8s.gcr.io/ingress-nginx/controller:v0.45.0 as a drop-in replacement)

[ericgraf]

*.rbd.csi.ceph.com storage class provisioners only supports Read/Write Single Pod while *.cephfs.csi.ceph.com supports Read/Write Multiple Pods .
https://kubernetes-csi.github.io/docs/drivers.html

I'm not sure what your use case is, whether you need a raw block volume or a filesystem volume, but if you need ReadWriteMany filesystem volume you will have deploy cephfs on top of ceph RBD.

[sanjeevSinha]

It is not the problem of Rook. You see that issue because PVC is Project/Namespace specific. And PV is available cluster wide accessible.
One has to create respective PVC for namespace where it is needed and not in another Namespace. PVC cannot be cross referenced in another Namespace.

[Jan Bebendorf]

It is not the problem of Rook. You see that issue because PVC is Project/Namespace specific. And PV is available cluster wide accessible. One has to create respective PVC for namespace where it is needed and not in another Namespace. PVC cannot be cross referenced in another Namespace.

That answer is just wrong and doesn't even make any sense at all. It has nothing todo with any namespaces and the OP already gave a hint on what's the issue.

As @ericgraf pointed out in his comment, ceph-csi doesn't support ReadWriteMany for rbd volumes (most likely because rbd doesn't support it, which would make absolute sense given that it's block storage). This is not a rook issue and it won't be fixed by ceph either so this issue can be closed.