mayastore without k8s
aep opened this issue · comments
we're moving away from ceph as a backing store for vms, due to poor nvme performance.
k8s only supports file volumes, but mayastore is a block storage.
would it be possible to setup maya without k8s?
if not, we'll investigage what can be reused at least for a new system
hmm k8s supports block volumes as well
Mayastor itself is not tightly coupled with k8s, we even have someone using us with most components deployed in k8s but the dataplane deployed as systemd services :)
Here's an example where we deploy most of our core services to a local docker:
https://github.com/openebs/mayastor-control-plane/tree/develop/deployer
Note we do have some components which are tied to k8s such as the upgrade through kubectl-plugin.
hmm k8s supports block volumes as well
oh. i must have misunderstood how openebs works.
does it have any other method than a PersistentVolumeClaim? which is always mounted afaik
with most components deployed in k8s but the dataplane deployed as systemd services :)
this sounds like a really good idea. best of both worlds. do we have any documents describing that?
Note we do have some components which are tied to k8s such as the upgrade through kubectl-plugin.
right, i dont mind learning k8s for operatiing mayastore itself. it makes sense for openebs to only support one thing as managment system. i just dont want to use it for the actual volumes because k8s does alot of things that are incompatible with hosting vms
hmm k8s supports block volumes as well
oh. i must have misunderstood how openebs works. does it have any other method than a PersistentVolumeClaim? which is always mounted afaik
The block volume is also bind mounted, as a block device!
PVC:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-1
spec:
accessModes:
- ReadWriteOnce
volumeMode: Block
resources:
requests:
storage: 10Gi
storageClassName: mayastor-nvmf
Pod:
kind: Pod
apiVersion: v1
metadata:
name: fio-$N
labels:
app: fio
spec:
volumes:
- name: ms-volume
persistentVolumeClaim:
claimName: pvc-1
containers:
- name: fio
image: mayadata/fio
args:
- sleep
- "1000000"
volumeDevices:
- devicePath: /dev/xvda
name: ms-volume
Here on pod /dev/xvda
is the block device!
with most components deployed in k8s but the dataplane deployed as systemd services :)
this sounds like a really good idea. best of both worlds. do we have any documents describing that?
We have a very simple example here, though specific for NixOS, for other linux flavours you'd have to see how to setup systemd service there and do equivalent.
https://github.com/openebs/mayastor/blob/develop/doc/run.md#running-as-a-systemd-service
this is super helpful, thank you so much. i will test this immediately.
while i have your attention, i noticed that mayastore uses nvme over tcp.
would you be ok with a patch that uses nvme over rdma?
would you be ok with a patch that uses nvme over rdma?
Absolutely :)