mayastore without k8s

[Arvid E. Picciani]

we're moving away from ceph as a backing store for vms, due to poor nvme performance.
k8s only supports file volumes, but mayastore is a block storage.
would it be possible to setup maya without k8s?

if not, we'll investigage what can be reused at least for a new system

[Tiago Castro]

hmm k8s supports block volumes as well

Mayastor itself is not tightly coupled with k8s, we even have someone using us with most components deployed in k8s but the dataplane deployed as systemd services :)
Here's an example where we deploy most of our core services to a local docker:
https://github.com/openebs/mayastor-control-plane/tree/develop/deployer
Note we do have some components which are tied to k8s such as the upgrade through kubectl-plugin.

[Arvid E. Picciani]

hmm k8s supports block volumes as well

oh. i must have misunderstood how openebs works.
does it have any other method than a PersistentVolumeClaim? which is always mounted afaik

with most components deployed in k8s but the dataplane deployed as systemd services :)

this sounds like a really good idea. best of both worlds. do we have any documents describing that?

Note we do have some components which are tied to k8s such as the upgrade through kubectl-plugin.

right, i dont mind learning k8s for operatiing mayastore itself. it makes sense for openebs to only support one thing as managment system. i just dont want to use it for the actual volumes because k8s does alot of things that are incompatible with hosting vms

[Tiago Castro]

hmm k8s supports block volumes as well

oh. i must have misunderstood how openebs works. does it have any other method than a PersistentVolumeClaim? which is always mounted afaik

The block volume is also bind mounted, as a block device!
PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-1
spec:
  accessModes:
   - ReadWriteOnce
  volumeMode: Block
  resources:
    requests:
      storage: 10Gi
  storageClassName: mayastor-nvmf

Pod:

kind: Pod
apiVersion: v1
metadata:
  name: fio-$N
  labels:
    app: fio
spec:
  volumes:
    - name: ms-volume
      persistentVolumeClaim:
       claimName: pvc-1
  containers:
    - name: fio
      image: mayadata/fio
      args:
        - sleep
        - "1000000"
      volumeDevices:
        - devicePath: /dev/xvda
          name: ms-volume

Here on pod /dev/xvda is the block device!

with most components deployed in k8s but the dataplane deployed as systemd services :)

this sounds like a really good idea. best of both worlds. do we have any documents describing that?

We have a very simple example here, though specific for NixOS, for other linux flavours you'd have to see how to setup systemd service there and do equivalent.
https://github.com/openebs/mayastor/blob/develop/doc/run.md#running-as-a-systemd-service

[Arvid E. Picciani]

this is super helpful, thank you so much. i will test this immediately.

while i have your attention, i noticed that mayastore uses nvme over tcp.
would you be ok with a patch that uses nvme over rdma?

[Tiago Castro]

would you be ok with a patch that uses nvme over rdma?

Absolutely :)