osbuild / osbuild

Build-Pipelines for Operating System Artifacts

Home Page:https://www.osbuild.org

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

mounts/devices don't allow actual disk topology confusing some tools

dustymabe opened this issue · comments

The CoreOS teams are writing a stage for bootupd for osbuild. If bootupd is targetting an environment with BIOS then it will use grub2-install to install to the MBR on the disk. grub2-install needs both access to the /boot filesystem as well as the root of the disk (for MBR writing). Up until now our bootupd stage has looked something like this:

      - type: org.osbuild.bootupd                                                                                     
        options:                                                                                                      
          bios:                                                                                                       
            disk: wholedisk                                                                                           
          static-configs: true                                                                                        
        devices:                                                                                                      
          wholedisk:                                                                                                  
            type: org.osbuild.loopback                                                                                
            options:                                                                                                  
              filename: disk.img                                                                                      
          efi:                                                                                                        
            type: org.osbuild.loopback                                                                                
            options:                                                                                                  
              filename: disk.img                                                                                      
              start:                                                                                                  
                mpp-format-int: '{image.layout[''EFI-SYSTEM''].start}'                                                
              size:                                                                                                   
                mpp-format-int: '{image.layout[''EFI-SYSTEM''].size}'                                                 
          boot:                                                                                                       
            type: org.osbuild.loopback                                                                                
            options:                                                                                                  
              filename: disk.img                                                                                      
              start:                                                                                                  
                mpp-format-int: '{image.layout[''boot''].start}'                                                      
              size:                                                                                                   
                mpp-format-int: '{image.layout[''boot''].size}'                                                       
          root:                                                                                                       
            type: org.osbuild.loopback                                                                                
            options:                                                                                                  
              filename: disk.img                                                                                      
              start:                                                                                                  
                mpp-format-int: '{image.layout[''root''].start}'                                                      
              size:                                                                                                   
                mpp-format-int: '{image.layout[''root''].size}'                                                       
        mounts:                                                                                                       
          - name: root                                                                                                
            type: org.osbuild.xfs                                                                                     
            source: root                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
            target: /                                                                                                 
          - name: boot                                                                                                
            type: org.osbuild.ext4                                                                                                                                                                                                          
            source: boot                                                                                              
            target: /boot                                                                                             
          - name: efi                                                                                                 
            type: org.osbuild.fat                                                                                     
            source: efi                                                                                               
            target: /boot/efi                                                                                         

So wholedisk is the target for the MBR install and the boot mount is where grub2-install is instructed to install the modules and everything needed for stage2.

This strategy, along with the work over in #1493, works. I'm able to install a system and it boots. I was pretty happy with this, but then I ran our CI tests (yay for CI!), and realized that any tests that we have that reboot the system yielded a system that doesn't come back after reboot.

This is because on CoreOS we have a unit that randomizes the UUID of our boot and root filesystems on the first boot (this ensures that every install of CoreOS in the field will have a unique UUID for those filesystems).

After that happens the system won't boot any longer because it seems to be strongly tied to the original UUID of the filesystem at the time grub2-install was run.

I did some more digging into this and found when running in osbuild the grub2-install ends up hitting slightly different logic. Running grub2-install with --verbose inside of an osbuild run ends up with some output that looks like:

grub2-install: info: /run/osbuild/osbuild-dev-mdwnfjue/loop2 is not present.                                          
grub2-install: info: Looking for /run/osbuild/osbuild-dev-mdwnfjue/loop2.                                             
grub2-install: info: /run/osbuild/osbuild-dev-mdwnfjue/loop2 is a parent of /run/osbuild/osbuild-dev-mdwnfjue/loop2.                                                                                                                        
grub2-install: info: /run/osbuild/osbuild-dev-mdwnfjue/loop2 is present.                                              
grub2-install: info: Looking for /run/osbuild/osbuild-dev-mdwnfjue/loop2.                                             
grub2-install: info: /run/osbuild/osbuild-dev-mdwnfjue/loop2 is a parent of /run/osbuild/osbuild-dev-mdwnfjue/loop2.                                                                                                                        
grub2-install: info: /run/osbuild/osbuild-dev-mdwnfjue/loop2 is present.                                              
grub2-install: info: Looking for /run/osbuild/osbuild-dev-mdwnfjue/loop2.                                             
grub2-install: info: /run/osbuild/osbuild-dev-mdwnfjue/loop2 is a parent of /run/osbuild/osbuild-dev-mdwnfjue/loop2.                                                                                                                        
grub2-install: info: drive = 0.                                                                                       
grub2-install: info: the size of hostdisk//run/osbuild/osbuild-dev-mdwnfjue/loop2 is 786432.                          
grub2-install: info: no partition map found for hostdisk//run/osbuild/osbuild-dev-mdwnfjue/loop2.                                                                                                                                           
grub2-install: info: /dev/loop0 is not present.                                                                       
grub2-install: info: Looking for /dev/loop0.                                                                          
grub2-install: info: /dev/loop0 is a parent of /dev/loop0.                                                            
grub2-install: info: /dev/loop0 is present.                                                                           
grub2-install: info: Looking for /dev/loop0.                                                                          
grub2-install: info: /dev/loop0 is a parent of /dev/loop0.                                                            
grub2-install: info: /dev/loop0 is present.                                                                           
grub2-install: info: Looking for /dev/loop0.                                                                          
grub2-install: info: /dev/loop0 is a parent of /dev/loop0.                                                            
grub2-install: info: 'grub-mkimage' '--directory' '/usr/lib/grub/i386-pc' '--prefix' '/grub2' '--output' '/run/osbuild/mounts/boot/grub2/i386-pc/core.img' '--dtb' '' '--sbat' '' '--format' 'i386-pc' '--compression' 'auto' '--appended-signature-size' '0' '--config' '/run/osbuild/mounts/boot/grub2/i386-pc/load.cfg' 'mdraid1x' 'ext2' 'biosdisk' 'search_fs_uuid'                                                                                                                 

whereas if I just run grub2-install on a booted system the logic is slightly different:

grub2-install: info: /dev/vda3 is not present.                                                                        
grub2-install: info: Looking for /dev/vda3.                                                                           
grub2-install: info: /dev/vda is a parent of /dev/vda3.                                                               
grub2-install: info: /dev/vda3 starts from 264192.                                                                                                                                                                                                                                                                                                                                                                                                                                       
grub2-install: info: opening the device hostdisk//dev/vda.                                                            
grub2-install: info: drive = 0.                                                                                       
grub2-install: info: the size of hostdisk//dev/vda is 20971520.                                                                                                                                                                             
grub2-install: info: drive = 0.                                                                                       
grub2-install: info: the size of hostdisk//dev/vda is 20971520.                                                       
grub2-install: info: Scanning for DISKFILTER devices on disk hostdisk//dev/vda.                                       
grub2-install: info: Scanning for mdraid1x devices on disk hostdisk//dev/vda.                                         
grub2-install: info: Scanning for mdraid09_be devices on disk hostdisk//dev/vda.                                      
grub2-install: info: Scanning for mdraid09 devices on disk hostdisk//dev/vda.                                         
grub2-install: info: Scanning for dmraid_nv devices on disk hostdisk//dev/vda.                                        
grub2-install: info: Scanning for ldm devices on disk hostdisk//dev/vda.                                                                                                                                                                                                                                                                                                                                                                                                                 
grub2-install: info: scanning hostdisk//dev/vda for LDM.                                                              
grub2-install: info: no LDM signature found.                                                                                                                                                                                                
grub2-install: info: Scanning for lvm devices on disk hostdisk//dev/vda.                                              
grub2-install: info: no LVM signature found.                                                                                                                                                                                                                                                                                                                                                                                                                                             
...                                                                                                                   
...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
...                                                                                                                   
grub2-install: info: Partition 0 starts from 2048.                                                                                                                                                                                          
grub2-install: info: Partition 1 starts from 4096.                                                                    
grub2-install: info: Partition 2 starts from 264192.                                                                                                                                                                                                                                                                                                                                                                                                                                     
grub2-install: info: drive = 0.                                                                                       
grub2-install: info: the size of hostdisk//dev/vda is 20971520.                                                       
grub2-install: info: /dev/vda is present.                                                                             
grub2-install: info: Looking for /dev/vda.                                                                            
grub2-install: info: /dev/vda is a parent of /dev/vda.                                                                                                                                                                                      
grub2-install: info: /dev/vda is present.                                                                                                                                                                                                   
grub2-install: info: Looking for /dev/vda.                                                                            
grub2-install: info: /dev/vda is a parent of /dev/vda.                                                                                                                                                                                                                                                                                                                                                                                                                                   
grub2-install: info: /dev/vda is present.                                                                                                                                                                                                   
grub2-install: info: Looking for /dev/vda.                                                                            
grub2-install: info: /dev/vda is a parent of /dev/vda.                                                                
grub2-install: info: 'grub-mkimage' '--directory' '/usr/lib/grub/i386-pc' '--prefix' '(,gpt3)/grub2' '--output' '/boot/grub2/i386-pc/core.img' '--dtb' '' '--sbat' '' '--format' 'i386-pc' '--compression' 'auto' '--appended-signature-size' '0' 'mdraid1x' 'ext2' 'part_gpt' 'biosdisk'                                                                                                                                                                                                

As you can see the grub2-mkimage call has an extra --config /run/osbuild/mounts/boot/grub2/i386-pc/load.cfg and the contents of that file look like:

[core@cosa-devsh ~]$ cat /boot/grub2/i386-pc/load.cfg                                                                 
search.fs_uuid 96d15588-3596-4b3c-adca-a2ff7279ea63 root                                                              
set prefix=($root)'/grub2'                                                                                            

This is because in the osbuild case the grub2 code detects that the MBR and stage 2 are on separate disks (i.e. the drive = 0 and drive = 1 output in the logs above) and thus determines it needs to leave a stage 1.5 hint about where the stage 2 is.

I think specifically it's hitting this code:

      if ((disk_module && grub_strcmp (disk_module, "biosdisk") != 0)                                                 
          || grub_drives[1]                                                                                           
          || (!install_drive                                                                                          
              && platform != GRUB_INSTALL_PLATFORM_POWERPC_IEEE1275)                                                  
          || (install_drive && !is_same_disk (grub_drives[0], install_drive))                                         
          || !have_bootdev (platform))                                                                                
        {                                                                                                             

so the (install_drive && !is_same_disk (grub_drives[0], install_drive)) means it drops into the following code and drops the load.cfg with the UUID binding.

What I would like to propose is for us to be able to pass in the whole disk in devices with partscan turned on via an option and then be able to specify our mounts with a parent device and a filesystem label or uuid to use for mounting. Something like this:

      - type: org.osbuild.bootupd
        options:
          bios:
            disk: wholedisk
          static-configs: true
        devices:
          wholedisk:
            type: org.osbuild.loopback
            options:
              filename: disk.img
        mounts:
          - name: root
            type: org.osbuild.xfs
            parent: wholedisk
            label: root
            target: /
          - name: boot
            type: org.osbuild.ext4
            parent: wholedisk
            label: boot
            target: /boot
          - name: efi
            type: org.osbuild.fat
            parent: wholedisk
            label: EFI-SYSTEM
            target: /boot/efi

This may seem unsafe, since many filesystems on a host can have the same FS label, but if we limit the consideration for this code to just the specified parent device then it should be fine.

This would enable things like loop0p1, loop0p2, loop0p3, etc.. and then the grub2-install code can detect and see that loop0p3 is a part of loop0 and do the right thing.

I'm not sure I follow the issue completely.

The CoreOS teams are writing a stage for bootupd for osbuild. If bootupd is targetting an environment with BIOS then it will use grub2-install to install to the MBR on the disk. grub2-install needs both access to the /boot filesystem as well as the root of the disk (for MBR writing). Up until now our bootupd stage has looked something like this:

      - type: org.osbuild.bootupd
        options:
          bios:
            disk: wholedisk
          static-configs: true
        devices:
          wholedisk:
            type: org.osbuild.loopback
            options:
              filename: disk.img
          efi:
            type: org.osbuild.loopback
            options:
              filename: disk.img
              start:
                mpp-format-int: '{image.layout[''EFI-SYSTEM''].start}'
              size:
                mpp-format-int: '{image.layout[''EFI-SYSTEM''].size}'
          boot:
            type: org.osbuild.loopback
            options:
              filename: disk.img
              start:
                mpp-format-int: '{image.layout[''boot''].start}'
              size:
                mpp-format-int: '{image.layout[''boot''].size}'
          root:
            type: org.osbuild.loopback
            options:
              filename: disk.img
              start:
                mpp-format-int: '{image.layout[''root''].start}'
              size:
                mpp-format-int: '{image.layout[''root''].size}'
        mounts:
          - name: root
            type: org.osbuild.xfs
            source: root
            target: /
          - name: boot
            type: org.osbuild.ext4
            source: boot
            target: /boot
          - name: efi
            type: org.osbuild.fat
            source: efi
            target: /boot/efi

So wholedisk is the target for the MBR install and the boot mount is where grub2-install is instructed to install the modules and everything needed for stage2.

This strategy, along with the work over in #1493, works. I'm able to install a system and it boots. I was pretty happy with this, but then I ran our CI tests (yay for CI!), and realized that any tests that we have that reboot the system yielded a system that doesn't come back after reboot.

This is because on CoreOS we have a unit that randomizes the UUID of our boot and root filesystems on the first boot (this ensures that every install of CoreOS in the field will have a unique UUID for those filesystems).

After that happens the system won't boot any longer because it seems to be strongly tied to the original UUID of the filesystem at the time grub2-install was run.

I did some more digging into this and found when running in osbuild the grub2-install ends up hitting slightly different logic. Running grub2-install with --verbose inside of an osbuild run ends up with some output that looks like:

grub2-install: info: /run/osbuild/osbuild-dev-mdwnfjue/loop2 is not present.
grub2-install: info: Looking for /run/osbuild/osbuild-dev-mdwnfjue/loop2.
grub2-install: info: /run/osbuild/osbuild-dev-mdwnfjue/loop2 is a parent of /run/osbuild/osbuild-dev-mdwnfjue/loop2.
grub2-install: info: /run/osbuild/osbuild-dev-mdwnfjue/loop2 is present.
grub2-install: info: Looking for /run/osbuild/osbuild-dev-mdwnfjue/loop2.
grub2-install: info: /run/osbuild/osbuild-dev-mdwnfjue/loop2 is a parent of /run/osbuild/osbuild-dev-mdwnfjue/loop2.
grub2-install: info: /run/osbuild/osbuild-dev-mdwnfjue/loop2 is present.
grub2-install: info: Looking for /run/osbuild/osbuild-dev-mdwnfjue/loop2.
grub2-install: info: /run/osbuild/osbuild-dev-mdwnfjue/loop2 is a parent of /run/osbuild/osbuild-dev-mdwnfjue/loop2.
grub2-install: info: drive = 0.
grub2-install: info: the size of hostdisk//run/osbuild/osbuild-dev-mdwnfjue/loop2 is 786432.
grub2-install: info: no partition map found for hostdisk//run/osbuild/osbuild-dev-mdwnfjue/loop2.
grub2-install: info: /dev/loop0 is not present.
grub2-install: info: Looking for /dev/loop0.
grub2-install: info: /dev/loop0 is a parent of /dev/loop0.
grub2-install: info: /dev/loop0 is present.
grub2-install: info: Looking for /dev/loop0.
grub2-install: info: /dev/loop0 is a parent of /dev/loop0.
grub2-install: info: /dev/loop0 is present.
grub2-install: info: Looking for /dev/loop0.
grub2-install: info: /dev/loop0 is a parent of /dev/loop0.
grub2-install: info: 'grub-mkimage' '--directory' '/usr/lib/grub/i386-pc' '--prefix' '/grub2' '--output' '/run/osbuild/mounts/boot/grub2/i386-pc/core.img' '--dtb' '' '--sbat' '' '--format' 'i386-pc' '--compression' 'auto' '--appended-signature-size' '0' '--config' '/run/osbuild/mounts/boot/grub2/i386-pc/load.cfg' 'mdraid1x' 'ext2' 'biosdisk' 'search_fs_uuid'

whereas if I just run grub2-install on a booted system the logic is slightly different:

grub2-install: info: /dev/vda3 is not present.
grub2-install: info: Looking for /dev/vda3.
grub2-install: info: /dev/vda is a parent of /dev/vda3.
grub2-install: info: /dev/vda3 starts from 264192.
grub2-install: info: opening the device hostdisk//dev/vda.
grub2-install: info: drive = 0.
grub2-install: info: the size of hostdisk//dev/vda is 20971520.
grub2-install: info: drive = 0.
grub2-install: info: the size of hostdisk//dev/vda is 20971520.
grub2-install: info: Scanning for DISKFILTER devices on disk hostdisk//dev/vda.
grub2-install: info: Scanning for mdraid1x devices on disk hostdisk//dev/vda.
grub2-install: info: Scanning for mdraid09_be devices on disk hostdisk//dev/vda.
grub2-install: info: Scanning for mdraid09 devices on disk hostdisk//dev/vda.
grub2-install: info: Scanning for dmraid_nv devices on disk hostdisk//dev/vda.
grub2-install: info: Scanning for ldm devices on disk hostdisk//dev/vda.
grub2-install: info: scanning hostdisk//dev/vda for LDM.
grub2-install: info: no LDM signature found.
grub2-install: info: Scanning for lvm devices on disk hostdisk//dev/vda.
grub2-install: info: no LVM signature found.
...
...
...
grub2-install: info: Partition 0 starts from 2048.
grub2-install: info: Partition 1 starts from 4096.
grub2-install: info: Partition 2 starts from 264192.
grub2-install: info: drive = 0.
grub2-install: info: the size of hostdisk//dev/vda is 20971520.
grub2-install: info: /dev/vda is present.
grub2-install: info: Looking for /dev/vda.
grub2-install: info: /dev/vda is a parent of /dev/vda.
grub2-install: info: /dev/vda is present.
grub2-install: info: Looking for /dev/vda.
grub2-install: info: /dev/vda is a parent of /dev/vda.
grub2-install: info: /dev/vda is present.
grub2-install: info: Looking for /dev/vda.
grub2-install: info: /dev/vda is a parent of /dev/vda.
grub2-install: info: 'grub-mkimage' '--directory' '/usr/lib/grub/i386-pc' '--prefix' '(,gpt3)/grub2' '--output' '/boot/grub2/i386-pc/core.img' '--dtb' '' '--sbat' '' '--format' 'i386-pc' '--compression' 'auto' '--appended-signature-size' '0' 'mdraid1x' 'ext2' 'part_gpt' 'biosdisk'

As you can see the grub2-mkimage call has an extra --config /run/osbuild/mounts/boot/grub2/i386-pc/load.cfg and the contents of that file look like:

[core@cosa-devsh ~]$ cat /boot/grub2/i386-pc/load.cfg
search.fs_uuid 96d15588-3596-4b3c-adca-a2ff7279ea63 root
set prefix=($root)'/grub2'

This is because in the osbuild case the grub2 code detects that the MBR and stage 2 are on separate disks (i.e. the drive = 0 and drive = 1 output in the logs above) and thus determines it needs to leave a stage 1.5 hint about where the stage 2 is.

I think specifically it's hitting this code:

      if ((disk_module && grub_strcmp (disk_module, "biosdisk") != 0)
          || grub_drives[1]
          || (!install_drive
              && platform != GRUB_INSTALL_PLATFORM_POWERPC_IEEE1275)
          || (install_drive && !is_same_disk (grub_drives[0], install_drive))
          || !have_bootdev (platform))
        {

so the (install_drive && !is_same_disk (grub_drives[0], install_drive)) means it drops into the following code and drops the load.cfg with the UUID binding.

Is this coming from grub2 itself or is it maybe something that bootupd is controlling?
I'm not seeing drive = 1 anywhere and wondering if this is in fact what's happening or if it's coming from somewhere else? Looking at the two logs, they're certainly not identical, but the parent device detection seems to be very similar.

What I would like to propose is for us to be able to pass in the whole disk in devices with partscan turned on via an option and then be able to specify our mounts with a parent device and a filesystem label or uuid to use for mounting. Something like this:

      - type: org.osbuild.bootupd
        options:
          bios:
            disk: wholedisk
          static-configs: true
        devices:
          wholedisk:
            type: org.osbuild.loopback
            options:
              filename: disk.img
        mounts:
          - name: root
            type: org.osbuild.xfs
            parent: wholedisk
            label: root
            target: /
          - name: boot
            type: org.osbuild.ext4
            parent: wholedisk
            label: boot
            target: /boot
          - name: efi
            type: org.osbuild.fat
            parent: wholedisk
            label: EFI-SYSTEM
            target: /boot/efi

This may seem unsafe, since many filesystems on a host can have the same FS label, but if we limit the consideration for this code to just the specified parent device then it should be fine.

This would enable things like loop0p1, loop0p2, loop0p3, etc.. and then the grub2-install code can detect and see that loop0p3 is a part of loop0 and do the right thing.

Is the right thing in this case "not writing the cfg with the uuid"?
We could certainly attach the whole device with partscan if it solves the issue. Your point about conflicting fs labels is a little worrying though. How do we limit it?

Is this coming from grub2 itself or is it maybe something that bootupd is controlling?
I'm not seeing drive = 1 anywhere and wondering if this is in fact what's happening or if it's coming from somewhere else? Looking at the two logs, they're certainly not identical, but the parent device detection seems to be very similar.

It is coming from grub2-install itself. Sorry, I snipped some of the log and the drive = 1 part got missed:

grub2-install: info: Opening dest `hostdisk//dev/loop0'.                                                                                   
grub2-install: info: drive = 1.                                                                                                            
grub2-install: info: the size of hostdisk//dev/loop0 is 20971520.                                                                          
grub2-install: info: /run/osbuild/osbuild-dev-mdwnfjue/loop2 is present.                                                                   
grub2-install: info: Looking for /run/osbuild/osbuild-dev-mdwnfjue/loop2.                                                                  
grub2-install: info: /run/osbuild/osbuild-dev-mdwnfjue/loop2 is a parent of /run/osbuild/osbuild-dev-mdwnfjue/loop2.                       
grub2-install: info: /run/osbuild/osbuild-dev-mdwnfjue/loop2 is present.                                                                   
grub2-install: info: Looking for /run/osbuild/osbuild-dev-mdwnfjue/loop2.                                                                  
grub2-install: info: /run/osbuild/osbuild-dev-mdwnfjue/loop2 is a parent of /run/osbuild/osbuild-dev-mdwnfjue/loop2.                       
grub2-install: info: drive = 0.                                                                                                            
grub2-install: info: the size of hostdisk//run/osbuild/osbuild-dev-mdwnfjue/loop2 is 786432.                                               
grub2-install: info: guessed root_dev `hostdisk//run/osbuild/osbuild-dev-mdwnfjue/loop2' from dir `/run/osbuild/mounts/boot/grub2/i386-pc'.
grub2-install: info: setting the root device to `hostdisk//run/osbuild/osbuild-dev-mdwnfjue/loop2'.                                        

Is the right thing in this case "not writing the cfg with the uuid"?

I think so. The /boot/grub2/i386-pc/load.cfg is something we don't really consider when considering the factors involved in bootloader updates and I think it complicates things to have it exist.

We could certainly attach the whole device with partscan if it solves the issue. Your point about conflicting fs labels is a little worrying though. How do we limit it?

My solution to that was to enlist a parent attribute to the mount specification that tells the mounting code to only consider partitions on this parent device when trying to find a filesystem to mount. So in the case of:

      - type: org.osbuild.bootupd     
        options:                      
          bios:                       
            disk: wholedisk           
          static-configs: true        
        devices:                      
          wholedisk:                  
            type: org.osbuild.loopback
            options:                  
              filename: disk.img      
        mounts:                       
          - name: root                
            type: org.osbuild.xfs     
            parent: wholedisk         
            label: root               
            target: /                 
          - name: boot                
            type: org.osbuild.ext4    
            parent: wholedisk         
            label: boot               
            target: /boot             
          - name: efi                 
            type: org.osbuild.fat     
            parent: wholedisk         
            label: EFI-SYSTEM         
            target: /boot/efi         

The devices code would set up a loopback mount on disk.img (the full disk), which is named wholedisk in our config here, and then the partscan would add device entries that represent the partitions on that device. For illustrative purposes let's assume that wholedisk gets the loop0 loopback device and then loop0p1, loop0p2, loop0p3, and loop0p4 exist as well.

Then the mount code would lookup the parent (here specified as wholedisk) and find it is loop0 and then only look for a filesystem with label root on that loop0 device. It would then set source for that mount entry to /dev/loop0p4. It would then do the same for boot and EFI-SYSTEM.

The result is that the same mounts exist as they did before, but they are associated with a parent device, unlike before.

And, actually, I think this more succinct description of the mounts/devices is easier to consume/read.

FTR I was able to confirm that getting things into this environment/setup would cause grub2 to do the right thing. In an extremely hacky way I renamed the mounts and devices only for my bootupd stage and then wrote some code that hardcoded special behavior:

diff --git a/osbuild/mounts.py b/osbuild/mounts.py
index d22d0ee9..6b14c76b 100644
--- a/osbuild/mounts.py
+++ b/osbuild/mounts.py
@@ -59,6 +59,24 @@ class MountManager:
     def mount(self, mount: Mount) -> Dict:
 
         source = self.devices.device_abspath(mount.device)
+        print(f"ZZZ source is {source} and mount.device is {mount.device.name}")
+
+        import stat
+        dn = os.path.dirname(source)
+        cwd = os.getcwd()
+        os.chdir(dn)
+        if mount.device.name == "bootupdroot":
+            source = os.path.join("/dev", "loop0p4")
+            os.mknod("loop0p4", mode=(stat.S_IMODE(0o600) | stat.S_IFBLK), device=os.makedev(259, 3))
+        elif mount.device.name == "bootupdboot":
+            source = os.path.join("/dev", "loop0p3")
+            os.mknod("loop0p3", mode=(stat.S_IMODE(0o600) | stat.S_IFBLK), device=os.makedev(259, 2))
+        elif mount.device.name == "bootupdefi":
+            source = os.path.join("/dev", "loop0p2")
+            os.mknod("loop0p2", mode=(stat.S_IMODE(0o600) | stat.S_IFBLK), device=os.makedev(259, 1))
+        os.chdir(cwd)
+        print(f"AAA source is {source} and mount.device is {mount.device.name}")
+
 
         root = os.fspath(self.root)
 

Then the environment looks like:

Pipeline raw-image: 1f5d854a07d019c32e13b55457681e5946898cd5ef70268757f0f1f7579bd327
Build
  root: <host>
  runner: org.osbuild.fedora38 (org.osbuild.fedora38)
org.osbuild.bootupd: 1f5d854a07d019c32e13b55457681e5946898cd5ef70268757f0f1f7579bd327 {
  "bios": {
    "disk": "wholedisk"
  },
  "static-configs": true
}
device/wholedisk (org.osbuild.loopback): loop0 acquired (locked: False)
device/bootupdefi (org.osbuild.loopback): loop1 acquired (locked: False)
device/bootupdboot (org.osbuild.loopback): loop2 acquired (locked: False)
device/bootupdroot (org.osbuild.loopback): loop3 acquired (locked: False)
ZZZ source is /run/osbuild/osbuild-dev-9f2806p8/loop3 and mount.device is bootupdroot
AAA source is /dev/loop0p4 and mount.device is bootupdroot
ZZZ source is /run/osbuild/osbuild-dev-9f2806p8/loop2 and mount.device is bootupdboot
AAA source is /dev/loop0p3 and mount.device is bootupdboot
ZZZ source is /run/osbuild/osbuild-dev-9f2806p8/loop1 and mount.device is bootupdefi
AAA source is /dev/loop0p2 and mount.device is bootupdefi
bash: cannot set terminal process group (1): Inappropriate ioctl for device
bash: no job control in this shell
bash-5.2# 
bash-5.2# 
bash-5.2# lsblk
NAME      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0       7:0    0   10G  0 loop 
├─loop0p1 259:0    0    1M  0 part 
├─loop0p2 259:1    0  127M  0 part /run/osbuild/mounts/boot/efi
├─loop0p3 259:2    0  384M  0 part /run/osbuild/mounts/boot
└─loop0p4 259:3    0    2G  0 part /run/osbuild/mounts
loop1       7:1    0  127M  0 loop 
loop2       7:2    0  384M  0 loop 
loop3       7:3    0    2G  0 loop 
zram0     251:0    0    8G  0 disk [SWAP]
vda       252:0    0   41G  0 disk 
├─vda1    252:1    0    1M  0 part 
├─vda2    252:2    0 1000M  0 part 
├─vda3    252:3    0  100M  0 part 
├─vda4    252:4    0    4M  0 part 
└─vda5    252:5    0 38.9G  0 part /run/osbuild/api/arguments
                                   /run/osbuild/inputs
                                   /run/osbuild/meta
                                   /run/osbuild/tree
                                   /proc/cmdline
                                   /etc/containers
                                   /var
                                   /usr
bash-5.2# ls /dev/loop*
/dev/loop0  /dev/loop0p2  /dev/loop0p3  /dev/loop0p4  /dev/loop1  /dev/loop2  /dev/loop3

and the logs from that run look like:

/usr/sbin/grub2-install: info: grub-mkimage --directory '/usr/lib/grub/i386-pc' --prefix '(,gpt3)/grub2' --output '/run/osbuild/mounts/boot/grub2/i386-pc/core.img'  --dtb '' --sbat '' --format 'i386-pc' -
-compression 'auto' --appended-signature-size 0   'mdraid1x' 'part_gpt' 'ext2' 'part_gpt' 'biosdisk'

Then the mount code would lookup the parent (here specified as wholedisk) and find it is loop0 and then only look for a filesystem with label root on that loop0 device. It would then set source for that mount entry to /dev/loop0p4. It would then do the same for boot and EFI-SYSTEM.

The result is that the same mounts exist as they did before, but they are associated with a parent device, unlike before.

And, actually, I think this more succinct description of the mounts/devices is easier to consume/read.

I wonder if it's simpler to rely on the partition ordering instead of labels here. osbuild knows the partition numbers, it created them itself. I think something like this might be more robust:

      - type: org.osbuild.bootupd
        options:
          bios:
            disk: wholedisk
          static-configs: true
        devices:
          wholedisk:
            type: org.osbuild.loopback
            options:
              filename: disk.img
        mounts:
          - name: root
            type: org.osbuild.xfs
            parent: wholedisk
            partition_number: 1
            target: /
          - name: boot
            type: org.osbuild.ext4
            parent: wholedisk
            partition_number: 2
            target: /boot
          - name: efi
            type: org.osbuild.fat
            parent: wholedisk
            partition_number: 3
            target: /boot/efi

I think having a partition_number that becomes mandatory when parent is used is easier to work with and define in schemas. Relying on labels would mean depending on a label being set during filesystem creation, which is generally optional, and happens separately from the usage of the mounts.

This isn't the most important part, so don't get me wrong, I think we should do this in general and I think it could simplify mounting in some cases where the "slice" (with start and size) device definition can be hard to read.