How to create a vmbuilder for debian
kenji21 opened this issue · comments
Wanted to try a debian using their https://cloud.debian.org/images/cloud/, but the ubuntu script download vmlinuz/initrd from unpacked cloud image, how to get them in order to create (and PR) a script for debian ?
can't get a netboot vm starting:
vmcli -c 4 -m 4096 -d disk.img --kernel netboot/debian-installer/arm64/linux --initrd netboot/debian-installer/arm64/initrd --cmdline="console=hvc0 irqfixup"
with kernel/initrd gathered from https://deb.debian.org/debian/dists/bullseye/main/installer-arm64/current/images/netboot/ (arm64 because on a M1 macbook)
Tried to mimic what is done in this Building a Debian Buster QEMU image for AARCH64 tutorial (but with bullseye), can't get vmcli to boot...
rebooted, rebuilt vmcli (using Xcode 13), now I got an error message:
vmcli -c 1 -m 4096 --kernel netboot/debian-installer/arm64/linux --initrd netboot/debian-installer/arm64/initrd --cmdline="console=hvc0 irqfixup"
Internal Virtualization error. The virtual machine failed to start.
OK, got it:
Note that Virtualization.framework provides all IO as virtio-pci, including the console (i.e. not a UART). The debian install kernel does not have virtio drivers, unfortunately. I ended up using debootstrap (--foreign) to install to a disc image on a Linux box... but I hear good things about Fedora etc.
Found the issue... if vmcli arch is x86_64 and you want a arm64 guest... it won't work
$ file build/vmcli
build/vmcli: Mach-O 64-bit executable arm64
$ git diff Makefile
diff --git a/Makefile b/Makefile
index 778fcbc..e0b86d7 100644
--- a/Makefile
+++ b/Makefile
@@ -8,7 +8,7 @@ build:
mkdir -p build
build/vmcli: build vmcli/Sources/vmcli/main.swift vmcli/Package.swift
- cd vmcli && swift build -c release --disable-sandbox
+ cd vmcli && swift build -c release --arch arm64 --disable-sandbox
cp vmcli/.build/release/vmcli build/vmcli
codesign -s - --entitlements vmcli/vmcli.entitlements build/vmcli
chmod +x build/vmcli
~/github/vmcli/build/vmcli -k debian-cloud/vmlinuz-5.10.0-9-arm64 --initrd=debian-cloud/initrd.img-5.10.0-9-arm64 -d debian-cloud/disk.raw --cmdline="console=tty0 console=hvc0 irqfixup root=/dev/vda1 ds=nocloud"
.success
[ OK ] Finished Load/Save Random Seed.
[ 1.588379] FAT-fs (vda15): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[ OK ] Mounted /boot/efi.
[ OK ] Reached target Local File Systems.
Starting Load AppArmor profiles...
Starting Create Volatile Files and Directories...
[ 1.622302] cryptd: max_cpu_qlen set to 1000
[ OK ] Finished Create Volatile Files and Directories.
Starting Update UTMP about System Boot/Shutdown...
[ OK ] Finished Update UTMP about System Boot/Shutdown.
[ 1.670484] audit: type=1400 audit(1626197364.804:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/sbin/chronyd" pid=271 comm="apparmor_parser"
[ 1.677001] audit: type=1400 audit(1626197364.812:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/man" pid=278 comm="apparmor_parser"
[ 1.677121] audit: type=1400 audit(1626197364.812:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_filter" pid=278 comm="apparmor_parser"
[ 1.677197] audit: type=1400 audit(1626197364.812:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_groff" pid=278 comm="apparmor_parser"
[ 1.682210] audit: type=1400 audit(1626197364.816:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=282 comm="apparmor_parser"
[ 1.682354] audit: type=1400 audit(1626197364.816:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=282 comm="apparmor_parser"
[ 1.688323] audit: type=1400 audit(1626197364.824:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lsb_release" pid=284 comm="apparmor_parser"
[ 1.698984] audit: type=1400 audit(1626197364.832:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="tcpdump" pid=286 comm="apparmor_parser"
[ OK ] Finished Load AppArmor profiles.
[ OK ] Finished Helper to synchronize boot up for ifupdown.
[ 1.815387] cloud-init[307]: Cloud-init v. 20.4.1 running 'init-local' at Tue, 13 Jul 2021 17:29:24 +0000. Up 1.80 seconds.
[ 1.819039] cloud-init[307]: 2021-07-13 17:29:24,956 - dmi.py[WARNING]: did not find either path /sys/class/dmi/id or dmidecode command
[ OK ] Finished Initial cloud-init job (pre-networking).
[ OK ] Reached target Network (Pre).
Starting Raise network interfaces...
[ OK ] Finished Raise network interfaces.
[ OK ] Reached target Network.
Starting Initial cloud-ini… (metadata service crawler)...
[ 19.671668] cloud-init[450]: Cloud-init v. 20.4.1 running 'init' at Tue, 13 Jul 2021 17:29:42 +0000. Up 19.66 seconds.
Glad you figured it out! You can alternatively just use the ubuntu kernel to boot debian if booting the debian kernel is non-trivial.
To have the debian kernel, I regenerated the initrd/vmlinuz with virtio support (it displays kernel output fine, but can't see the login prompt): https://intl.cloud.tencent.com/document/product/213/9929
I have created a script like the ubuntu vmbuilder script for creation of a debian VM:
https://gist.github.com/remyvv/62954ecb033fade09e4407fd9a80365e
The final trick was to add the virtio_console module to the initramfs, without recompiling the whole kernel.
This needs the cmdline to start with "console=tty0" before the "console=hvc0".
The early parts of the kernels output are send to tty0 this way (which is not visible to us), and when the initramfs is loaded the output is also send to the hvc0 output.
References used: