Failure to restore vm with disk size smaller than default

Question

Failure to restore vm with disk size smaller than default

a-barinov opened this issue 3 years ago · comments

Qubes OS release

4.1.0-rc3

Brief summary

When trying to restore a TemplateVM with root or private volume smaller than default (10G and 2G respectively), restore fails with error "not enough data".
I have not tested other vm types or backups created in Qubes 4.0.

Steps to reproduce

Create a new TemplateVM.
Shrink root volume to 2.5Gb and private volume to 0.5Gb, this requires a bit of manual fs, lvm and partition (in case of root volume) resizing.
Backup the VM.
Try restoring from the backup.

Expected behavior

VM getting restored from backup and can be started without errors.

Actual behavior

Trying to restore, you get the following error:

qubesadmin.backup: -> Restoring debian-full...
qubesadmin.backup: Extracting data: 1.8 GiB to restore
qubesadmin.backup: Failed to restore volume root of VM debian-full1: Data import failed: not enough data (copied 2684354560 bytes, expected 10737418240 bytes)
qubesadmin.backup: Failed to restore volume private of VM debian-full1: Data import failed: not enough data (copied 536870912 bytes, expected 2147483648 bytes)
qubesadmin.backup: -> Done.

Restored VM then cannot start.

Rusty Bird · Answer 1 · Sun Jan 02 2022 23:38:47 GMT+0800 (China Standard Time)

This line of the restore code should check for inequality
Storage driver support:
1. file-reflink: works
2. lvm_thin: see this comment
3. legacy file: looks similar to lvm_thin

Marek Marczykowski-Górecki · Answer 2 · Sun Jan 02 2022 23:49:40 GMT+0800 (China Standard Time)

qubesadmin.backup: Failed to restore volume root of VM debian-full1: Data import failed: not enough data (copied 2684354560 bytes, expected 10737418240 bytes)

~~This is 2.5G, while the root volume is 10G normally, so it isn't just about missing resize, it really got not enough data.~~
EDIT: not enough coffee...

Pierre Alain · Answer 3 · Tue Apr 05 2022 22:04:44 GMT+0800 (China Standard Time)

I ran into the same type of issue with a template build for a mirageOS unikernel (qubes-mirage-firewall). My builder.conf have TEMPLATE_ROOT_SIZE ?= 50MiB which should be enough for the unikernel and I can see in the build-logs/template-mirage.log:

...
--> Creating template config file...
--> Linking /home to /rw/home...
--> Linking /usr/local to /rw/usrlocal...
Reducing image size (calling cleanup_image)...
9268 -rw-r--r-- 1 user user 52428800 Apr  5 09:53 qubeized_images/mirage-firewall/root.img
--> Cleaning up image file...
--> Compacting image file...
mnt: 40 MiB (41944064 bytes) trimmed
9204 -rw-r--r-- 1 user user 52428800 Apr  5 09:53 qubeized_images/mirage-firewall/root.img
--> Unmounting qubeized_images/mirage-firewall/root.img
Qubeized image stored at: qubeized_images/mirage-firewall/root.img
...

and when I try to install it into dom0:

...
mirage-firewall: Importing data
Traceback (most recent call last):
  File "/bin/qvm-template-postprocess", line 5, in <module>
    sys.exit(main())
  File "/usr/lib/python3.8/site-packages/qubesadmin/tools/qvm_template_postprocess.py", line 431, in main
    loop.run_until_complete(post_install(args))
  File "/usr/lib64/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/lib/python3.8/site-packages/qubesadmin/tools/qvm_template_postprocess.py", line 297, in post_install
    import_root_img(vm, args.dir)
  File "/usr/lib/python3.8/site-packages/qubesadmin/tools/qvm_template_postprocess.py", line 107, in import_root_img
    vm.volumes['root'].import_data(stream=tar.stdout)
  File "/usr/lib/python3.8/site-packages/qubesadmin/storage.py", line 270, in import_data
    self._qubesd_call('Import', payload_stream=stream)
  File "/usr/lib/python3.8/site-packages/qubesadmin/storage.py", line 76, in _qubesd_call
    return self.app.qubesd_call(
  File "/usr/lib/python3.8/site-packages/qubesadmin/app.py", line 730, in qubesd_call
    return self._parse_qubesd_response(stdout)
  File "/usr/lib/python3.8/site-packages/qubesadmin/base.py", line 109, in _parse_qubesd_response
    raise exc_class(format_string, *args)
qubesadmin.exc.QubesException: Data import failed: not enough data (copied 52428800 bytes, expected 10737418240 bytes)
warning: %post(qubes-template-mirage-firewall-4.0.6-202204051348.noarch) scriptlet failed, exit status 1
...

The root.img size seems to be the same from the logfile and in dom0, but the "< default=10G" size trigger the issue.

EDIT: add the full backtrace

Pierre Alain · Answer 4 · Tue Apr 05 2022 22:41:34 GMT+0800 (China Standard Time)

Not sure if that can help, and my knowledge does not permit me to try that patch, but as we seem to have the root.img size in qvm_template_postprocess.py:def import_root_img(vm, source_dir), can it be possible to change (https://github.com/QubesOS/qubes-core-admin-client/blob/400ea49e84bfba30f50b658ca0b23e4bb18c52f2/qubesadmin/tools/qvm_template_postprocess.py#L119)?

        with subprocess.Popen(['tar', 'xSOf', root_path + '.tar'],
                stdout=subprocess.PIPE) as tar:
-            vm.volumes['root'].import_data(stream=tar.stdout)
+            vm.volumes['root'].import_data_with_size(stream=tar.stdout, root_size) 
        if tar.returncode != 0:
            raise qubesadmin.exc.QubesException('root.img extraction failed')

And also at https://github.com/QubesOS/qubes-core-admin-client/blob/400ea49e84bfba30f50b658ca0b23e4bb18c52f2/qubesadmin/tools/qvm_template_postprocess.py#L134

Rusty Bird · Answer 5 · Thu Apr 07 2022 01:06:27 GMT+0800 (China Standard Time)

@palainp Neat! import_data_with_size() instead of the current resize() + import_data() would work even for storage drivers that don't support shrinking.

qubesadmin.backup.restore has the relevant code, but there's lots of indirection so it might have to be refactored a bit.

Alex · Answer 6 · Mon May 02 2022 03:21:16 GMT+0800 (China Standard Time)

OK, I was kind of able to fix this, but due to too many API layers this is not the cleanest fix.

Contrary to what @rustybird said previously, the error message is actually triggered in admin.vm.volume.Import script, section starting from line 69. Commenting it out will get rid of the error, but your restored volumes will not be resized. I therefore decided to look for a cleaner way which sent me down a rabbit hole...

First, inspired by @palainp I made a similar change to restore code in restore.py

+++ restore.py  2022-04-30 20:04:24.962001232 +0200
@@ -1866,7 +1866,7 @@
     def _handle_volume_data(self, vm, volume, stream):
         '''Wrap volume data import with logging'''
         try:
-            volume.import_data(stream)
+            volume.import_data_with_size(stream=stream, size=volume.size)
         except Exception as err:  # pylint: disable=broad-except
             self.log.error('Failed to restore volume %s of VM %s: %s',
                 volume.name, vm.name, err)

This was not enough though, as for whatever reason a function just below patched prohibits volume downsizing during restore, which I consider to be a bug. So one additional fix is needed:

+++ restore.py  2022-04-30 20:04:24.962001232 +0200
@@ -1874,8 +1874,7 @@
     def _handle_volume_size(self, vm, volume, size):
         '''Wrap volume resize with logging'''
         try:
-            if volume.size < size:
-                volume.resize(size)
+            volume.resize(size)
         except Exception as err:  # pylint: disable=broad-except
             self.log.error('Failed to resize volume %s of VM %s to %d: %s',
                 volume.name, vm.name, size, err)

This fixes restore code to properly take care of volume downsizing. I was hoping that this will do the job, but turned out the underlying storage driver cannot downsize volumes! I think this was done to prevent user from reducing volume size from qube settings or commend line (which is a right thing to do), but baking this into storage backend does sounds way too extreme to me. Anyway, I'm not keen to edit underlying Qubes API so I decided to fully restore the ability to downsize volumes.

As I use LVM backend, all the changes I needed are limited to lvm.py.

First, I had to remove backend restriction to downsize volumes:

+++ lvm.py  2022-04-30 21:00:51.311000681 +0200
@@ -607,12 +607,12 @@
             msg = 'Can not resize reađonly volume {!s}'.format(self)
             raise qubes.storage.StoragePoolException(msg)
 
-        if size < self.size:
-            raise qubes.storage.StoragePoolException(
-                'For your own safety, shrinking of %s is'
-                ' disabled (%d < %d). If you really know what you'
-                ' are doing, use `lvresize` on %s manually.' %
-                (self.name, size, self.size, self.vid))
 
         if size == self.size:
             return

Second, we need to allow extend operation to downsize volumes:

+++ lvm.py  2022-04-30 21:00:51.311000681 +0200
@@ -766,7 +766,7 @@
                    '--virtualsize=' + cmd[3] + 'B', '--', cmd[1]]
     elif action == 'extend':
         assert len(cmd) == 3, 'wrong number of arguments for extend'
-        lvm_cmd = ["lvextend", "--size=" + cmd[2] + 'B', '--', cmd[1]]
+        lvm_cmd = ["lvresize", "--force", "--size=" + cmd[2] + 'B', '--', cmd[1]]
     elif action == 'activate':
         assert len(cmd) == 2, 'wrong number of arguments for activate'
         lvm_cmd = ['lvchange', '--activate=y', '--', cmd[1]]

Keep in mind this will likely enable system-wide LVM volumes downsize. A cleaner approach would be to modify Qubes internal API so that restore operation could indicate that volume downsize shall be allowed. Alternatively, all the downsize checks can be move to appropriate frontends, leaving my added backend downsize capability intact. I definitely prefer second solution as more architecturally correct ne.

Demi Marie Obenour · Answer 7 · Mon May 02 2022 08:11:46 GMT+0800 (China Standard Time)

@a-barinov You, I, and @marmarek all agree that the downsize checks do not belong in qubesd. Care to send a PR with your changes?

Alex · Answer 8 · Tue May 03 2022 13:05:17 GMT+0800 (China Standard Time)

You'll have it in a few days. Let me verify that other frontends have appropriate downsize checks, and also check other backends.

qubesos-bot · Answer 9 · Fri Apr 28 2023 00:02:12 GMT+0800 (China Standard Time)

Automated announcement from builder-github

The package core-admin-client has been pushed to the r4.2 testing repository for the Debian template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing bullseye-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade