unable to detect active ESP

Question

unable to detect active ESP

firasuke opened this issue a year ago · comments

Hey there,

I am attempting to create a live installation ISO for a custom distribution I am making. I have a rootfs setup, with a bootloader configured (syslinux for mbr/bios, and grub for EFI). I am using Arch Linux kernel and modules (since booster works just fine with Arch), and I am using booster universal to generate an initramfs image for my custom distribution.

Regardless of the mode I boot in under QEMU, and regardless of what I use syslinux vs GRUB I am receiving the following error:

booster: unable to detect active ESP: open /sys/firmware/efi/efivars/LoaderDevicePartUUID-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f: no such file or directory

I have stumbled upon the following issue #149 which seems similar, and a patch was proposed their to remove this check (as my custom distribution is close in structure to Alpine, so it made sense). After trying the patch I am left with the following error booster: /dev/sr0 unknown drive type which might be related to the device manager I am using mdevd, but then again shouldn't booster run before any service/device manager has been started? or should there be hooks for the device manager like udev?

It seems that booster is trying to load efivarfs which is working, but then is trying to open a non-existent directory? I thought booster handled the mounting of sys? If so why is it failing? And why is it trying to detect an ESP when in MBR/BIOS mode? Also, could it be relying on the system's service boot scripts to mount sys or for more functionality? If so, shouldn't booster not do that because it comes before in the boot graph order?

Anatol Pomozov · Answer 1 · Sat Jul 08 2023 00:49:44 GMT+0800 (China Standard Time)

Thanks for the report. I am happy to debug this problem. Is there a way to reproduce the failing case with QEMU? I am using Arch and QEMU+failing image would be the best option I think.

Or another option is to enable debug logs with booster.log=debug,console and then post its output here.

Firas Khalil Khana · Answer 2 · Sat Jul 08 2023 02:15:42 GMT+0800 (China Standard Time)

Alright I added booster.log=debug,console to kernel cmdline and I got the following additional information:

root= param is not specified. Use GPT partition autodiscovery with guid type 4f68bce3-e8cd-4db1-96e7-fbcaf984b709

booster: unable to detect active ESP: open /sys/firmware/efi/efivars/LoaderDevicePartUUID-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f: no such file or directory

Anatol Pomozov · Answer 3 · Sat Jul 08 2023 02:44:19 GMT+0800 (China Standard Time)

And what is the debug output in case of booster: /dev/sr0 unknown drive type error?

Anatol Pomozov · Answer 4 · Sat Jul 08 2023 02:45:40 GMT+0800 (China Standard Time)

So it looks like you do not have a root= boot param, so booster switches into GPT autodiscovery mode. Is there a reason the bootloader (syslinux and grub) do not specify root= param?

Firas Khalil Khana · Answer 5 · Sat Jul 08 2023 02:55:50 GMT+0800 (China Standard Time)

Here's what I have in my grub.cfg (using GRUB for EFI):

...
    linux  /boot/vmlinuz modules=loop,squashfs,erofs rw booster.log=debug,console
    initrd /boot/initramfs
...

and in my syslinux.cfg (using ISOLINUX for BIOS):

    KERNEL /boot/vmlinuz
    INITRD /boot/initramfs
    APPEND rw booster.log=debug,console

Anatol Pomozov · Answer 6 · Sat Jul 08 2023 02:58:22 GMT+0800 (China Standard Time)

So you have 2 options here

specify root=/dev/XXX
rely on GPT autodiscovery. And you need the efi module loaded and fully functional.

The later case won't work for non-EFI configuration though.

Firas Khalil Khana · Answer 7 · Sat Jul 08 2023 03:21:08 GMT+0800 (China Standard Time)

So you have 2 options here
* specify `root=/dev/XXX`

* rely on GPT autodiscovery. And you need the `efi` module loaded and fully functional.
The later case won't work for non-EFI configuration though.

In the case of an ISO, the root filesystem will be stored mostly on /dev/sr0, so I passed root=/dev/sr0 and booster throws a huge log, with udev doing detections and stuff, and then it throws:

...
booster: /dev/sr0 unknown drive type
...

should i try adding rootfstype? I am also pretty sure the loop module is being added to booster..

Anatol Pomozov · Answer 8 · Sat Jul 08 2023 03:27:28 GMT+0800 (China Standard Time)

Please add root=/dev/sr0 and the debug parameters. And then post the debug logs.

Firas Khalil Khana · Answer 9 · Sat Jul 08 2023 04:04:58 GMT+0800 (China Standard Time)

I am unable to capture the output of QEMU even with -nographics and with -D or -serial...

Anatol Pomozov · Answer 10 · Sat Jul 08 2023 05:53:00 GMT+0800 (China Standard Time)

Try -nographic -display none maybe?

Anatol Pomozov · Answer 11 · Sun Jul 09 2023 02:58:08 GMT+0800 (China Standard Time)

Also do you have a reproduction insructions for the image you are making? Ideally all the cases that can break need to be part of booster's integration tests here https://github.com/anatol/booster/tree/master/tests.

Firas Khalil Khana · Answer 12 · Sun Jul 09 2023 23:40:17 GMT+0800 (China Standard Time)

Try -nographic -display none maybe?

Nope, that did not work... I will upload screenshots showcasing most of the error log.

Also do you have a reproduction insructions for the image you are making? Ideally all the cases that can break need to be part of booster's integration tests here https://github.com/anatol/booster/tree/master/tests.

I am creating an ISO for a custom Linux distribution I am working on. I have a root filesystem consisting mainly of musl libc and toybox for user space and some other utilities.

I have a boot directory where I copy Arch's kernel as vmlinuz, and its modules to usr/lib/modules.

I am then creating a universal booster initramfs image by running:

booster \
    build \
    --force \
    --compression=zstd \
    --config=booster.yaml \
    --universal \
    --strip \
    rootfs/boot/initramfs

Here's what I have inside booster.yaml

network:
  dhcp: on
universal: true
modules: loop,squashfs,erofs
compression: zstd
strip: true

Here's what I have inside boot/grub/grub.cfg:

...
    linux  /boot/vmlinuz modules=loop,squashfs,erofs rw booster.log=debug,console
    initrd /boot/initramfs
...

Here's what I have inside boot/syslinux/syslinux.cfg:

    KERNEL /boot/vmlinuz
    INITRD /boot/initramfs
    APPEND rw booster.log=debug,console

I am using the following instructions to create the ISO:

grub-mkrescue -o mycustom.iso rootfs

I also use xorriso:

xorriso -as mkisofs \
    -iso-level 3 \
    -full-iso9660-filenames \
    -rational-rock \
    -eltorito-boot boot/syslinux/isolinux.bin \
    -eltorito-catalog boot/syslinux/boot.cat \
    -no-emul-boot -boot-load-size 4 -boot-info-table \
    -isohybrid-mbr rootfs/boot/syslinux/isohdpfx.bin \
    -output mycustom.iso rootfs/

When I run the ISO, here's the error I get in EFI mode:

qemu-system-x86_64 -enable-kvm -cpu host -boot d -cdrom mycustom.iso -m 1G -bios /usr/share/ovmf/x64/OVMF.fd

and in regular BIOS/mbr mode:

qemu-system-x86_64 -enable-kvm -cpu host -boot d -cdrom mycustom.iso -m 1G

Firas Khalil Khana · Answer 13 · Mon Jul 10 2023 00:16:00 GMT+0800 (China Standard Time)

It seems the error is as follows:

found a new device /dev/sr0
blkinfo for /dev/sr0: type=mbr UUID=00000000 LABEL=
booster: /dev/sr0 unknown drive type
/dev/sr0 unknown drive type

Apparently it is failing to detect /dev/sr0 correctly, as it is of type iso9660 and not mbr? Also is there a way to specify a certain LABEL or UUID (perhaps the same one that the ISO was created with -volid to make it easier to auto-detect it for this use case?

Anatol Pomozov · Answer 14 · Mon Jul 10 2023 03:50:37 GMT+0800 (China Standard Time)

Apparently it is failing to detect /dev/sr0 correctly, as it is of type iso9660 and not mbr?

Indeed, this one of the problems.

Another problem is that SMART tries to read data from /dev/sr0 and fails and it is where the unknown drive type comes from.

Thank you for the provided instructions. I added an integration test for *.iso that show the error. And handled the error case. The changes are pushed to issue-223 branch. Please pull it, compile the init, and pack it with your *.iso. Let me know the results.

Firas Khalil Khana · Answer 15 · Mon Jul 10 2023 04:16:33 GMT+0800 (China Standard Time)

Apparently it is failing to detect /dev/sr0 correctly, as it is of type iso9660 and not mbr?

Indeed, this one of the problems.

Another problem is that SMART tries to read data from /dev/sr0 and fails and it is where the unknown drive type comes from.

Thank you for the provided instructions. I added an integration test for *.iso that show the error. And handled the error case. The changes are pushed to issue-223 branch. Please pull it, compile the init, and pack it with your *.iso. Let me know the results.

Alright, thanks for your work on this.

I tested the new branch, and unfortunately with EFI (grub) it is still showing the same error as above, but with syslinux (mbr/BIOS mode) it is failing but with some difference in the error log:

found a new device /dev/sr0
blkinfo for /dev/sr0: type=mbr UUID=44523d4f LABEL=
booster: /dev/sr0 unknown drive type
/dev/sr0 unknown drive type

Again is mbr the correct filesystem/type for this? Also shouldn't it be mounted somewhere (/new_root) before being switch_rooted into?

A friend of mine wrote this explanation article on how this is supposed to work.

Anatol Pomozov · Answer 16 · Mon Jul 10 2023 10:43:45 GMT+0800 (China Standard Time)

blkinfo for /dev/sr0: type=mbr UUID=44523d4f LABEL= it looks like you still use the old booster. With the new change the filesystem will be detected as iso9660.

Please make sure you pulled issue-223 compiled it, replaced the existing init and then regenerate the image for iso.

Firas Khalil Khana · Answer 17 · Wed Jul 12 2023 03:00:33 GMT+0800 (China Standard Time)

blkinfo for /dev/sr0: type=mbr UUID=44523d4f LABEL= it looks like you still use the old booster. With the new change the filesystem will be detected as iso9660.

Please make sure you pulled issue-223 compiled it, replaced the existing init and then regenerate the image for iso.

Ok I am no longer facing the previous error, instead I am now facing this:

mount(/dev/sr0): read-only file system

Anatol Pomozov · Answer 18 · Wed Jul 12 2023 03:15:29 GMT+0800 (China Standard Time)

It is great to hear about the progress. To handle the latter problem please use ro boot parameter instead of rw. The kernel assumes cdrom is a read-only device and thus requires read-only filesystem.

Firas Khalil Khana · Answer 19 · Wed Jul 12 2023 03:27:32 GMT+0800 (China Standard Time)

It is great to hear about the progress. To handle the latter problem please use ro boot parameter instead of rw. The kernel assumes cdrom is a read-only device and thus requires read-only filesystem.

Ok this is awesome! It is attempting to search for init which is /sbin/init. My custom distribution does not have /sbin and the init is located over at /usr/bin/init.

So I specified init=/usr/bin/init and it worked!

Firas Khalil Khana · Answer 20 · Wed Jul 12 2023 03:28:36 GMT+0800 (China Standard Time)

I guess the service manager at this point should handle the rw remount, as the current filesystem is read-only.

Firas Khalil Khana · Answer 21 · Wed Jul 12 2023 04:44:27 GMT+0800 (China Standard Time)

Also suppose the case the root filesystem was compressed using squashfs or EROFS, would it be possible to include support for this as well? Or perhaps just simply modprobeing the modules erofs and squashfs would suffice?

Anatol Pomozov · Answer 22 · Wed Jul 12 2023 04:50:32 GMT+0800 (China Standard Time)

I am not much familiar with iso handling. But if you provide instructions on how such erofs/squashfs is provisioned, then I can add an integration test to make sure booster handles it as well.

Firas Khalil Khana · Answer 23 · Wed Jul 12 2023 13:56:13 GMT+0800 (China Standard Time)

Thanks for your time and effort!

I will do some more testing, and open a new issue if I encounter another edge case

Anatol Pomozov · Answer 24 · Thu Jul 13 2023 00:28:14 GMT+0800 (China Standard Time)

Sounds good!

And if you provide me instructions how to create an erofs/squashfs ISO file I'll add it to the booster's integration tests.

Firas Khalil Khana · Answer 25 · Wed Jul 19 2023 20:39:44 GMT+0800 (China Standard Time)

Ok, consider the case the root filesystem was compressed using squashfs or erofs into a file called rootfs.erofs.

According to this, I think booster should check whether the root filesystem is compressed/squashed or not, if it is then an additional mount command should be run:

...
mount -t erofs -o ro /dev/some_loop_device /some_mount_directory
...

So in addition to checking for /dev/sr0 and mounting it, booster should also check whether another parameter is passed that specifies the location of the root filesystem on /dev/sr0 whether it was compressed or not (for example another root= as an argument for booster and not for cmdline)..