Support for Docker Desktop on Mac (Apple Silicon)
tiagonix opened this issue · comments
Hello,
This project is fantastic! It is precisely what I'm looking for! Which is a small Ceph deployment with S3 support to experiment locally. I plan to replace MinIO with picoceph.
I can run picoceph in Ubuntu machines, amd64 (QEMU), and aarch64 (Parallels).
I'm using Parallels on Mac to install Ubuntu (and the docker.io Debian package) and then run picoceph within the VM (aarch64). I used Parallels as a test base to see if I could run picoceph on the Mac machine, and it worked!
But I need to run it straight on Mac using Docker Desktop without relying on Parallels. I plan to use Docker Compose as well. Since the Docker Desktop on Mac uses Virtual Machines behind the scenes, and it worked on Parallels (a VM), it should work—the issue I'm facing is with the NBD; could picoceph use a loopback device instead?
So, when I try to run it straight on Mac, here's the error:
➜ ~ docker run --rm --name picoceph -p7480:7480 ghcr.io/bucket-sailor/picoceph:latest
time=2024-02-20T23:26:00.073Z level=INFO msg="Creating ceph directories"
time=2024-02-20T23:26:00.073Z level=INFO msg="Writing ceph.conf"
time=2024-02-20T23:26:00.073Z level=INFO msg="Configuring manager"
time=2024-02-20T23:26:00.073Z level=INFO msg="Configuring monitor"
time=2024-02-20T23:26:00.073Z level=INFO msg="Creating OSD device"
time=2024-02-20T23:26:00.074Z level=INFO msg="Configuring RADOS Gateway"
time=2024-02-20T23:26:00.081Z level=ERROR msg="Could not run picoceph" error="could not create OSD device: could not load nbd module: exit status 1: "
I need to become more familiar with Mac internals. I researched online and couldn't find any information about how to fix this. I also read a message on Reddit stating that it should be possible, but I need help locating instructions.
Is this an issue with picoceph that perhaps needs to add support for Docker Desktop on Macs, or is it outside its scope?
NOTE: The Docker Desktop is running on my Mac; I can start other Docker containers as usual.
Thanks!
Just a quick update! I changed it to use loopback instead of NBD:
diff --git a/cmd/main.go b/cmd/main.go
index e5707b3..11f79c4 100644
--- a/cmd/main.go
+++ b/cmd/main.go
@@ -343,35 +343,27 @@ func createOSDDevice(ctx context.Context) error {
}
// Create a qemu image.
- cmd = exec.CommandContext(ctx, "qemu-img", "create", "-f", "qcow2", "/var/lib/ceph/disk/osd-0.qcow2", "10G")
+ cmd = exec.CommandContext(ctx, "qemu-img", "create", "-f", "raw", "/var/lib/ceph/disk/osd-0.raw", "10G")
if output, err := cmd.Output(); err != nil {
return fmt.Errorf("could not create qemu image: %w: %s", err, output)
}
- // Mount the image using nbd.
- cmd = exec.CommandContext(ctx, "/sbin/modprobe", "nbd")
- if output, err := cmd.Output(); err != nil {
- return fmt.Errorf("could not load nbd module: %w: %s", err, output)
- }
-
- // Find the next free nbd device.
- nbdDevicePath, err := findNextFreeNBDDevice()
+ // Attach the image to a loop device.
+ cmd = exec.CommandContext(ctx, "losetup", "-f", "--show", "/var/lib/ceph/disk/osd-0.raw")
+ loopDevicePath, err := cmd.Output()
if err != nil {
- return fmt.Errorf("could not find free nbd device: %w", err)
+ return fmt.Errorf("could not attach image to loop device: %w: %s", err, string(loopDevicePath))
}
+ loopDevice := strings.TrimSpace(string(loopDevicePath))
- cmd = exec.CommandContext(ctx, "qemu-nbd", "--connect="+nbdDevicePath, "/var/lib/ceph/disk/osd-0.qcow2")
- if output, err := cmd.Output(); err != nil {
- return fmt.Errorf("could not mount qemu image: %w: %s", err, output)
- }
-
- cmd = exec.CommandContext(ctx, "pvcreate", nbdDevicePath)
+ // Use loopDevice in the following commands instead of NBD device path.
+ cmd = exec.CommandContext(ctx, "pvcreate", loopDevice)
cmd.Env = append(os.Environ(), "DM_DISABLE_UDEV=1")
if output, err := cmd.Output(); err != nil {
return fmt.Errorf("could not create physical volume: %w: %s", err, output)
}
- cmd = exec.CommandContext(ctx, "vgcreate", "ceph-vg", nbdDevicePath)
+ cmd = exec.CommandContext(ctx, "vgcreate", "ceph-vg", loopDevice)
cmd.Env = append(os.Environ(), "DM_DISABLE_UDEV=1")
if output, err := cmd.Output(); err != nil {
return fmt.Errorf("could not create volume group: %w: %s", err, output)
@@ -383,6 +375,8 @@ func createOSDDevice(ctx context.Context) error {
return fmt.Errorf("could not create logical volume: %w: %s", err, output)
}
+ // Remember to detach the loop device when done (not shown here).
+
return nil
}
And I used earthly to rebuild it. It still works! Now I have to figure out how to try it on aarch64.
Thanks for your thorough testing, good to hear it's working on a variety of platforms.
I'll take a look at getting the OSX support working, Docker Desktop for Mac includes NBD which I want to continue using (because of the thin provisioning qemu-nbd provides). I think the main change I'll have to make is that instead of getting the user to bind mount /dev and /lib/modules, I can use nsenter from the privileged container to access the parents mount space. Should improve the ergonomics on linux aswell.
@dpeckett, thank you for this awesome project! You rock! ^_^
I'm not using the above changes related to loopback, as you mentioned, Docker Desktop for Mac does indeed have NBD built in, but I didn't know that before. lol
The thing is, the following code block fails because there isn't any /lib/modules/* in there:
- // Mount the image using nbd.
- cmd = exec.CommandContext(ctx, "/sbin/modprobe", "nbd")
- if output, err := cmd.Output(); err != nil {
- return fmt.Errorf("could not load nbd module: %w: %s", err, output)
- }
-
So I removed it to test, and it proceeded!
But then it failed further down the process at the lvcreate step, which is the lack of dm_mod. But it was fixed today in the Docker Desktop for Mac 4.28.0. I'll be testing it soon. Both loopback and NBD approach were failing now at the same spot.
If it works, it would be a matter of trying to detect if the nbd is already available, likely by verifying if the files /dev/nbd* exist, meaning the nbd is already loaded or is built-in, or looking under /sys (no sure yet about the best way of detecting nbd's presence). Then, skip the modprobe.
Awesome work man, thanks for chasing up the dm_mod situation. I'll be very keen to see this running cleanly on an M1 (I don't know too many ways to run ceph on M1s today).
I've just pushed an update that will make the modprobe step non-fatal and will perform a check for nbd support before continuing, perhaps we can add a loop device fallback at some point but meh nbd support is pretty ubiquitous.
I think picoceph could be a real boon to the ceph project tbh, reducing the barrier to entry is always a huge benefit.
If you don't mind sharing I'd love if you could drop me an email about what you are working on, feedback from users really helps.
Wheee! It's working!
The minimum requirement is Docker Desktop for Mac 4.28.0.
Thank you, Damian!
Cheers!