ex-integrity.composefs: Tracking issue
cgwalters opened this issue · comments
composefs/ostree (and beyond)
Background
A key design goal of ostree at its creation was to not require any new functionality in the Linux kernel. The baseline mechanisms of hard links and read-only bind mounts suffice to manage views of read-only filesystem trees.
However, for Docker and then podman, overlayfs was created to more efficiently support copy-on-write semantics - also crucially, overlayfs is a layered filesystem; it can work with any underlying (modern) Linux filesystem as a backend.
More recently, composefs was created which builds on overlayfs with more integrity features. This tracking issue is for the integration of composefs and ostree.
System integrity
ostree does not provide significant support for truly immutable system state; a simple mount -o remount,rw /usr will allow direct persistent modification of the underlying files.
There is ostree fsck, but this is inefficient and manual, and further still today does not cover the checked-out deployment roots (so e.g. newly added binaries in the deployment root aren't found).
Accidental damage protection
It is important to ostree to support "user owns machine" scenarios, where the user is root on their own computer and must have the ability to make persistent changes.
But it's still useful to have stronger protection against accidental damage. Due to the way composefs works using fs-verity, a simple mount -o remount,rw can no longer silently modify files. First, the mounted composefs is always read-only; there is no write support in composefs. Access to the distinct underlying persistent root filesystem can be more strongly separated and isolated.
Support for "sealed" systems
It's however also desirable to support a scenario where an organization wants to produce computing devices that are "sealed" to run only code produced (or signed) by that organization. These devices should not support persistent unsigned code.
ostree does not have strong support for this model today, and composefs should fix it.
Phase 0: Basic integration (experimental)
In this phase, we will land an outstanding pull request which adds basic integration that enables booting a system using composefs as a root filesystem. In this phase, a composefs image is dynamically created on the client using the ostree metadata.
This has already led us to multiple systems integration issues. So far, all tractable.
A good milestone to mark completion of this phase is landing a CI configuration to ostree which builds and deploys a system using composefs, and verifies it can be upgraded.
In this phase, there is no direct claimed support for "sealed" systems (i.e. files are not necessarily signed).
Phase 1: Basic rootfs sealing (experimental)
In this phase, support for signatures covering the composefs is added. A key question to determine is when the composefs file format is stable. Because the PR up until this point defaults to "re-synthesizing" the composefs on the client, the client must reproduce exactly what was generated server side and signed.
Phase 2: Secure Boot chaining (experimental)
This phase will document how to create a complete system using Secure Boot which chains to a root filesystem signature using composefs.
This may also depend on #2753 and #1951
Here is a sketch for how we can support trusted boot using composefs and fs-verity signatures.
During build:
- Generate a public/private key pair
- Copy the public key into the new rootfs for the commit (e.g.
/etc/pki/fsverity/cfs.pub) - During initrd generation in the rootfs, pass
--install /etc/pki/fsverity/cfs.pubto dracut, which will copy the public key into the initrd. - Add a module to dracut that loads the public key into the fs-verity keyring (see https://gitlab.com/CentOS/automotive/rpms/dracut-fsverity for an example)
- Generate a UKI or aboot image containing the above initrd, the kernel and kernel command line. The kernel command line uses a generic
ostree=latestargument, because at this point we don't know the final deployment id. See also discussion in #2844 - Sign the UKI with a private key that is trusted by your secureboot keyring.
- Stuff the UKI into the rootfs next to the normal kernel
- Save the rootfs as objects in the ostree repo, giving the digest of the rootdir
- ostree commit normally just stores the above digest in the metadata. But now we also take the private key from step 1 (passed as argument to
ostree commit) and generate a composefs image file based on the rootdir digest. We sign this file with the private key and store the signature as extra metadata in the commit object. - The entire commit object is GPG signed and pushed to a repo.
During install:
- Pull a new commit
- Verify the GPG signature
- When deploying we look at the metadata from the commit, in particular the rootdir digest and the signature. The rootdir digest (and the repo objects) is used to construct a new composefs file, and the signature is used to sign the composefs image file when enabling fs-verity on it.
- The BLS files are created in /boot/loader.[01] that points to the deploy dir with the composefs file, and the /boot/loader symlink is atomically switched to new loader.[01] dir. This BLS file contains the deploy id we deployed into in the kernel
ostree=...arg. - The UKI is put somewhere where the boot loader can find it. (EFI partition, aboot partition, etc)
During boot:
- The firmware loads the UKI, verifies it according to the secureboot keyring and boots kernel+initrd.
- The initrd mounts the disk partitions.
- The initrd notices the kernel arg "ostree=latest" and looks for the BLS file in /boot/loader with index=1 (i.e. most recent deploy, or index=2 if we're in fallback mode).
- The initrd parses the BLS file, which contains the full ostree=... argument. This lets us find the deploy directory (like
/ostree/deploy/fedora-coreos/deploy/443ae0cd86a7dd4c6f5486a2283471b3c8f76fc5dcc4766cf935faa24a9e3d34.0). (Note at this point that we can't trust either the BLS file or the deploy dir.) - The initrd loads /etc/pki/fsverity/cfs.pub into the kernel keyring for fs-verity. (Trusted, as its in signed initrd.)
- The initrd mounts the composefs with the
LCFS_MOUNT_FLAGS_REQUIRE_SIGNATUREflag. This ensures that the file to be mounted has a signature, and thus can only be read if the matching public key is loaded in the keyring. - On top of the composefs we bind mount writable things like /var and /etc.
- Pivot-root into the new composefs mount, which now will verify all further reads from the readonly parts of rootfs are valid.
Beyond
At this point, we should have gained significant experience with the system. We will determine when to mark this as officially stabilized after this.
Phase 3: "Native composefs"
Instead of "ostree using composefs", this proposes to flip things around, such that more code lives underneath the "composefs" project. A simple strawman proposal here is that we have the equivalent of ostree-prepare-root.service actually be composefs-prepare-root.service and live in github.com/containers/composefs.
Related issues:
Phase 4: Unified container and host systems
This phase builds on the native composefs for hosts and ensures that containers (e.g. podman) share backing storage with the host system and as much code as possible.
About the composefs file format stability. The plan is to guarantee stability in general, and there is way to change it by specifying a version when you generate the file. However, I don't want to give any stability guarantees until the overlay xattr changes has landed in the upstream kernel, because only then do we know they will not change.
Ok, I ran into a snag with this approach:
When doing an update, the new deploy is written, and when we enable fs-verity on it, with the signature, fs-verity fails. The reason is that the new signature is signed with the new certificate and the public key is not in the kernel keyring at the time of deploy.
We have a similar issue at image building time, where we would need to load the public key into the keyring of the host (i.e. build) machine.
It doesn't feel right to load any keys like this into the keyring at any time other then boot (and the keyrings are bound to be sealed anyway). So, I think we need to delay the application of the signature to the first boot, as we can then guarantee that the right keys are loaded.
This issue will be discussed this Friday at 9:00am EST in https://meet.jit.si/moderated/2e9be89e0e9ee06647b4719784578a6251f72eec9a07829bc9212e57c4883816
I wrote down some random ramblings about the Phase 3 approach to kickstart
the meeting/discussion:
Basic assumptions:
- All the tools for building images and handling images at build time
(rpm-ostree, ostree repos, etc) remain as is. All we change is how
the OS is deployed on the target machine. - We assume that an initrd is used
- We assume that an systemd is used
Points that needs consideration:
-
What mechanism/protocol is used to download a new update to
the target machine. -
What is the format of the data that is downloaded.
-
How does this format allow incremental downloads (i.e. avoiding
downloading parts already available locally). And to what degree is
it incremental? (i.e. just skip already available files, or full
binary delta). -
composefs will add support for installing images to a shared
/.composefs "repo", as well as some code similar to
ostree-prepare-root steps. But, who will be responsible for the
boot part of the deployment, like generating bls files, putting
initrds from the images in the right place, merging /etc, rollback,
etc. Its not clear where the border between composefs and ostree
lies. -
What other ostree system features do we support?
- overlay /usr?
- rpm layering?
- initrd regeneration (and/or injection) on host?
- local kernel cmdline changes
- anything else?
-
How do we handle signatures. My current work is based on fs-verity
signatures. So, we load a public key into the kernel fs-verity
keyring and then just validate at mount time that any fs-verity
signature exists, which will trigger validataion by the kernel.
This has some problems:- You can't sign the file at deploy time unless you load the public
key into the kernel keyring (which may be sealed) - You can't even look at the contents for any composefs image other
than the one you booted (at least when using per-build keys). - Fedora (for example) currently doesn't enable the fs-verity
signature support in the kernel.
Another approach is to store a regular x509 signature of the
fs-verity digest in a file, then we verify this signature in the
initrd, passing the signed digest when mounting the composefs
image.This means adding some crypto code to the initrd prepare-root code,
adding a dependency on openssl or similar. - You can't sign the file at deploy time unless you load the public
More recently, composefs was created which builds on overlayfs with more integrity features. This design document describes the high
Looks like this sentence is cut before the end (from the first comment)
The initrd notices the kernel arg "ostree=latest" and looks for the BLS file in /boot/loader with index=1 (i.e. most recent deploy, or index=2 if we're in fallback mode).
How do we know that we are in fallback mode?
But, who will be responsible for the boot part of the deployment, like generating bls files, putting initrds from the images in the right place, merging /etc, rollback, etc. Its not clear where the border between composefs and ostree lies.
This is done by ostree & rpm-ostree in Fedora CoreOS for example.
You can't even look at the contents for any composefs image other than the one you booted (at least when using per-build keys).
Isn't it possible to validate fs-verity signatures from userspace with requiring the key to be loaded in the kernel?
+1 from me for this approach in general. Thanks for writing it up!
The BLS files are created in /boot/loader.[01] that points to the deploy dir with the composefs file, and the /boot/loader symlink is atomically switched to new loader.[01] dir. This BLS file contains the deploy id we deployed into in the kernel ostree=... arg.
Note that in some cases (direct UEFI boot, with or without systemd-boot, with UKIs), there won't be BLS configs or they won't be used. #2753 (comment) as an alternative proposal to let the initrd figure out which entry was booted and which ostree deployment should be used by storing the ostree deployment hash in the filename of the UKI and then reading it from the EFI variables in the initrd.
You can't even look at the contents for any composefs image other than the one you booted (at least when using per-build keys).
Isn't it possible to validate fs-verity signatures from userspace with requiring the key to be loaded in the kernel?
The way fs-verity signatures work right now is that they are verified by the kernel automatically when you open the file.
If we had a standalone signature file paired with the non-signed composefs file we could do the validation in userspace like this. But if the composefs file was signed we can't even look at it until we've loaded the right key into the kernel.
Was the meeting recorded? I wanted to join but accidentally overslept.
Was the meeting recorded? I wanted to join but accidentally overslept.
Sorry, it wasn't. Probably should have. We decided to make this a recurring meeting, so there will be another one on Friday June 16 at the same time (9:30am EST).
I may also argue at some point that this should be a composefs meeting and not an ostree meeting and we'd do it alongside or in the github.com/containers context.
So, there is a keyctl_pkey_verify() syscall:
https://man7.org/linux/man-pages/man3/keyctl_pkey_sign.3.html
I think using this during mount to verify a signature file is much better and more flexible than using the built-in fs-verity signatures, because you can then both access the composefs image file without the key, and enable fs-verity on it without knowing the public key. I will have a look at adding support for this to libcomposefs.
5. Add
This issue will be discussed this Friday at 9:00am EST in https://meet.jit.si/moderated/2e9be89e0e9ee06647b4719784578a6251f72eec9a07829bc9212e57c4883816
Is there an .ics file, etc. for this meeting so I can add to my calendar?
Is there an .ics file, etc. for this meeting so I can add to my calendar?
I spent about 10 minutes doing a web search trying to figure out how to craft an ics file by hand with no luck...
I got one anyway thanks, pity it's not a really simple right-click export calendar event type of thing
I’d assume most email providers have that built-in? At least Gmail does. Or at least a workaround https://webapps.stackexchange.com/questions/114322/google-calendar-share-single-event-as-ics
OK I know I'm bouncing things around here; I want to take the subthread related to ostree from here containers/composefs#151 back here because it's about reusing ostree's existing signature infrastructure.
Here's my strawman...
- When generating an ostree commit, we also run
mkcomposefs --print-digest. That digest is embeded in the ostree commit in the inline metadata. (OK I think what we really want here isostree commit --embed-composefs-digestand the equivalent C API) - (we're having a discussion of format versions; this would need to appear probably as something like
ostree.composefs-digest.v0) - We change ostree-prepare-root to link to libostree (again)
- ostree-prepare-root verifies the target commit object using all of its existing signature infrastructure (note in this case, the keys are found exactly the same way they're found in the real root)
- ostree-prepare-root extracts the digest from the commit object and passes it to
mount.composefs - mount.composefs verifies the digest and mounts
Does this make sense?
OK, I started drafting a change which cleans up the metadata bits slightly (I think we want an API for this and not a repo flag, it's more flexible), and also starts dropping the current custom signature bits #2891 in preparation for switching to the above plan.
OK #2891 merged - the big next step is "change ostree-prepare-root to link to libostree" and I have some WIP on that but I think we need to do a release first.
Beyond that, with this signature plan some concerns were raised about dragging GPG into the initramfs. I think there are multiple answers to this:
- First, GPG can already be compile-time disabled for those who don't want it (and instead want the ed25519-libsodium bits)
- We could also try having our initramfs module just not include the dependencies, though this may involve fighting the system
- If necessary, we can also investigate splitting it out into a loadable
/usr/lib64/ostree/ostree-sign-gpg.so - Failing that...instead of linking ostree-prepare-root to libostree proper, we can just link to glib to start, and then pull in exactly just the required code for signature verification we want (this may be relatively tractable)
Finally, we can also look at supporting a custom basic signing flow (as was originally done here), although personally I would lean a lot more towards supporting a "standard" one in composefs over a new custom one for ostree.
I think we should try just linking prepare-root to glib, and then include in the required headers and c files from libostree/ directly into the binary. Seems least work and most likely to work.
Sometimes I wish initramfs was like a really simple persistent filesystem (like ext4/vfat or even smaller) with a throwaway overlayfs rather than a tmpfs... That way minimising initramfs would be less of a consideration...
Its not just GPG btw, other dependencies we may not want are: curl, libarchive, libssh,
Also, I don't think we can just disable GPG in the build. You may need GPG for e.g. flatpak in the system libostree, even though you don't use it in the initrd.
I think we should try just linking prepare-root to glib, and then include in the required headers and c files from libostree/ directly into the binary. Seems least work and most likely to work.
A less adhoc way to do that would be to split part of libostree into a static libotcore (or something) like:
noinst_LTLIBRARIES += libotcore.la
libotcore_la_SOURCES = <files split out from libostree_1_la_SOURCES>
libotcore_la_LIBADD = libglnx.la $(OT_INTERNAL_GIO_UNIX_LIBS)
libostree_1_la_LIBADD += libotcore.la
ostree_prepare_root_LDADD += libotcore.la
Sometimes I wish initramfs was like a really simple persistent filesystem (like ext4/vfat or even smaller) with a throwaway overlayfs rather than a tmpfs... That way minimising initramfs would be less of a consideration...
Nothing stops one from doing that at all; the initramfs can easily mount the target disk and load data from some other partition (or files stored in the ESP), and use that to mount the root. It just brings into question what tool manages that state...I think the inherent tradeoffs in adding a new thing aren't worth it.
@cgwalters we at Toradex are working on this on embedded systems (and by "this", I mean HAB with integrity checks through to the rootfs using ostree+composefs). Are you guys still having regular sync calls on this topic? We'd like to join if possible, and I think we can provide some insight into embedded use cases that could be helpful.
Hi everyone!
I used ostree config --repo=/ostree/repo set ex-integrity.composefs true but now on every command I receive this kind of error every time when I use ostree or rpm-ostree ostree config --repo=/ostree/repo set ex-integrity.composefs false error: opening repo: composefs required, but libostree compiled without support Is there any way to fix it? I am new to immutable Fedora
https://ostreedev.github.io/ostree/composefs/ this page sends to this issue.
@iho Just run ostree config --repo=/ostree/repo set ex-integrity.composefs false ...
@cgwalters we at Toradex are working on this on embedded systems (and by "this", I mean HAB with integrity checks through to the rootfs using ostree+composefs). Are you guys still having regular sync calls on this topic? We'd like to join if possible, and I think we can provide some insight into embedded use cases that could be helpful.
Definitely, though I think it's usually most efficient to have high level discussions asynchronously and reserve realtime discussion for resolving "contentious" or difficult issues, as we did with the signature support thread.
If you see something not convered, feel free to just comment here; if it's big enough we can open a dedicated tracking issue for it too.
Reminder the meeting is today at 3:30pm CEST, 9:30am EST in https://meet.jit.si/moderated/2e9be89e0e9ee06647b4719784578a6251f72eec9a07829bc9212e57c4883816
cc @giuseppe - can you attend?
yes, I will attend
(Sorry I missed the last meeting since I'm not quite good at oral english and missed the accurate time.
I could attend on demand if something is on me if needed.)
Sometimes I wish initramfs was like a really simple persistent filesystem (like ext4/vfat or even smaller) with a throwaway overlayfs rather than a tmpfs... That way minimising initramfs would be less of a consideration...
Nothing stops one from doing that at all; the initramfs can easily mount the target disk and load data from some other partition (or files stored in the ESP), and use that to mount the root. It just brings into question what tool manages that state...I think the inherent tradeoffs in adding a new thing aren't worth it.
So I couldn't help but try this, dunno if it's a wider thing for Fedora/CentOS Stream etc. But this could certainly prove useful for Automotive where boot time requirements are strict and more and more is being asked to be added to initramfs for early boot:
https://github.com/ericcurtin/initoverlayfs
It's not fully booting yet, must fix up some switchroot stuff, but the left are monotonic times for an initrd/initramfs and the right is monotonic times for an initoverlayfs:
Any progress here?
I got initoverlayfs booted without issue on Fedora (without ostree integration as of yet):
https://github.com/ericcurtin/initoverlayfs
The way it's implemented you can get faster boot times if you strip initramfs to just storage drivers, udev and a small C binary called pre-init in the repo, who's only role is to switch to initoverlayfs once the storage devices and drivers have been initialised and exec systemd. initoverlayfs contains all the contents of a "fat" initramfs. It means you can scale initoverlayfs to any size without degrading boot time:
if we did integrate this with ostree, it means you could consider even higher level abstractions in prepare-root than glib, like Rust and/or C++ without degrading boot time because of increased binary size.
@ericcurtin Personally I'd actually have tried to write this as one statically linked Rust binary, reimplementing the bit of udev we need, but anyways...let's track this as a separate issue?
Filed #3066
EDIT: to be clear, thanks for working on this!
@cgwalters Since the ostree repo can still be remounted r,w and therefore objects deleted or altered - and yes fs-verity will trigger a fault when accessing those files, aren't systems with ostree+composefs/fs-verity enabled still susceptible to denial of service attacks? more over if the functional flow of such a system depends on certain files being present (which usually means these files being accessible), by corrupting them after a deployment and rebooting we could be actually changing the system behavior.
just checking if am I missing something fundamental.
I am testing my deployments and it all works as expected btw (ostree+composefs+fs-verity)
aren't systems with ostree+composefs/fs-verity enabled still susceptible to denial of service attacks?
This is not an attack specific to composefs. See for example for Android using dm-verity:
- https://source.android.com/docs/security/features/verifiedboot/verified-boot#handling-verification-errors
- https://source.android.com/docs/security/features/verifiedboot/boot-flow#dm-verity-corruption
An approach would be to read / access the entire filesystem tree on boot (maybe in the background) to completely verify the content of the image. This however has a cost.
It will always be up to distributions to make sure that applications fail securely if a file is missing / an IO error is returned.
An approach would be to read / access the entire filesystem tree on boot (maybe in the background) to completely verify the content of the image. This however has a cost.
It will always be up to distributions to make sure that applications fail securely if a file is missing / an IO error is returned.
thanks @travier , yes that makes sense.
Since the ostree repo can still be remounted r,w and therefore objects deleted or altered - and yes fs-verity will trigger a fault when accessing those files, aren't systems with ostree+composefs/fs-verity enabled still susceptible to denial of service attacks?
Yes, but any sufficiently privileged code can also just open the raw disk device and delete the partition table too. This is a bit of a nuanced issue and it depends on the threat model. In the case of "container runtime breakout" it's possible that the container may have sufficient privileges to do "remount in host mount namespace, + rm", but not to open raw block devices - which can actually be gated more strictly by a targeted LSM/SELinux policy.
But if the case is "container breakout got kernel mode code execution" and there's just no inherent way to stop a DoS in that scenario.
@cgwalters Since the ostree repo can still be remounted r,w and therefore objects deleted or altered - and yes fs-verity will trigger a fault when accessing those files, aren't systems with ostree+composefs/fs-verity enabled still susceptible to denial of service attacks? more over if the functional flow of such a system depends on certain files being present (which usually means these files being accessible), by corrupting them after a deployment and rebooting we could be actually changing the system behavior. just checking if am I missing something fundamental.
I am testing my deployments and it all works as expected btw (ostree+composefs+fs-verity)
Note also if this results in boot failure, greenboot will rollback to last known healthy boot, now that boot could also get corrupted, but pretty much all verity techniques today are susceptible this sort of denial of service attack.
Even verity techniques that's don't rely on dm-verity/fs-verity such as Android Verified Boot and UKIs are also susceptible to this, change the bits in a UKI or Android Boot Image and the machine won't read/load it.
Yeah, even rolling back won't necessarily work on cases where you also want to protect against rollback attacks, so the only way would be to do similar as done by android, and reboot into a recovery mode of some sort (application / product specific).
Yeah, even rolling back won't necessarily work on cases where you also want to protect against rollback attacks, so the only way would be to do similar as done by android, and reboot into a recovery mode of some sort (application / product specific).
Even recovery mode can get corrupted, it's kinda a never ending chain.
One feature OSTree has is that if you want more than AB rollbacks, you can in theory have as many rollbacks as you want ABCD, but just AB is common.
WRT to fs-verity triggering on detected issues, anyone knows why the kernel doesnt implement the config/option to just trigger a reboot on detection? just something around these lines - might be a bit more complex but this would be the idea - in a configurable way
diff --git a/fs/overlayfs/namei.c b/fs/overlayfs/namei.c
index 80391c687c2ad..dbbec0a9c862c 100644
--- a/fs/overlayfs/namei.c
+++ b/fs/overlayfs/namei.c
@@ -991,8 +991,11 @@ int ovl_verify_lowerdata(struct dentry *dentry)
int err;
err = ovl_maybe_lookup_lowerdata(dentry);
- if (err)
+ if (err) {
+ if (err == -ENOENT)
+ BUG_ON(1);
return err;
+ }
return ovl_maybe_validate_verity(dentry);
}
diff --git a/fs/overlayfs/util.c b/fs/overlayfs/util.c
index 89e0d60d35b6c..fd039df0851d9 100644
--- a/fs/overlayfs/util.c
+++ b/fs/overlayfs/util.c
@@ -1309,6 +1309,7 @@ int ovl_validate_verity(struct ovl_fs *ofs,
&verity_algo, NULL);
if (digest_size == 0) {
pr_warn_ratelimited("lower file '%pd' has no fs-verity digest\n", datapath->dentry);
+ BUG_ON(1);
return -EIO;
}
@@ -1317,6 +1318,7 @@ int ovl_validate_verity(struct ovl_fs *ofs,
memcmp(metacopy_data.digest, actual_digest, xattr_digest_size) != 0) {
pr_warn_ratelimited("lower file '%pd' has the wrong fs-verity digest\n",
datapath->dentry);
+ BUG_ON(1);
return -EIO;
}
@ldts could be a useful feature. Somthing you are interested in hacking on @ldts ?
I think this would have to be dynamically switchable on/off, it's also dangerous to randomly power off sometimes.
But maybe if in the boot path it's the right thing to do for example (even then it may not be, depends on your use-case), but maybe not after that?
How do you alert the user of this problem is another thing.
sure I wouldnt mind. perhaps getting @alexlarsson input first since he extended overlayfs with this support. will follow up but it seems to me that for the use case - so must be configurable- where the full file system demands integrity would be the right thing to do
Might be worth considering if this would integrate with:
systemd-bsod
I don't really think that is a good approach. For example, in a safety situtation, you might have some really important process running, and then some unimportant process hits a fs-verity issue, rebooting the systemd and stopping the important process. You might also be able to misuse this as a form of attack. I.e. loopback mount a file with a known incorrect fs-verity data to reboot the system.
What might be more useful is to have the option of having the process issuing the failing operation get a signal that kills the process, say SIGBUS or something like that.
100%, that is a better design choice overall but wont that be some time away for most products? IMO is possible that some systems might want to chose a more conservative approach when it comes to security and completely shutdown no matter what any other process might be running: I mean, the system is compromised already....and then before kernel reboot/shutdown, something could be logged (maybe attach some form of kernel notifier) so that persistent storage (RPMB?) can be updated to flag the situation during reboot... I am thinking that perhaps a router would fit in that sort of product but I dont know for sure
I think you will have a hard time selling it upstream.
I think you will have a hard time selling it upstream.
yes I fully agree as well: but I think it is the sort of patch worth carrying off-tree
Well, that is up to whoever wants to carry it. I'm not very interested in that kind of thing though.
Hey, I'm wondering what's the current state of file verification? It's a bit hard to process all relevant threads as a project outsider.
In particular I'm trying to figure out whether IMA is working or not (#3240), or is it supposed to be replaced with composefs?
Most engineering effort is on composefs/erofs/fs-verity right now
one note, if using an old systemd (ie, 250 (250.5+) with systemd-boot and ostree+composefs, you might need this systemd patch to find out the boot/ EFI partition:
Subject: [PATCH] gpt: composefs: block device on sysroot
rootfs corresponds to the composefs overlay: use sysroot instead.
Signed-off-by: Jorge Ramirez-Ortiz <jorge@foundries.io>
---
src/gpt-auto-generator/gpt-auto-generator.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/gpt-auto-generator/gpt-auto-generator.c b/src/gpt-auto-generator/gpt-auto-generator.c
index 64ca9bb2f9..a7c564ca5c 100644
--- a/src/gpt-auto-generator/gpt-auto-generator.c
+++ b/src/gpt-auto-generator/gpt-auto-generator.c
@@ -774,7 +774,7 @@ static int add_mounts(void) {
* here. */
r = readlink_malloc("/run/systemd/volatile-root", &p);
if (r == -ENOENT) { /* volatile-root not found */
- r = get_block_device_harder("/", &devno);
+ r = get_block_device_harder("/sysroot", &devno);
if (r == -EUCLEAN)
return btrfs_log_dev_root(LOG_ERR, r, "root file system");
if (r < 0)
--
2.34.1
otherwise boot will not be mounted
@ldts Hmm, yes this relates to containers/composefs#280 as well as #3193
@ldts Hmm, yes this relates to containers/composefs#280 as well as #3193
the gpt generator on systemd-boot 250 looks for the block device (expecting it contains the boot partition to mount besides the rootfs partition) using "/" (which with composefs is actually the overlay). So switching it to sysroot seems a better choice when using ostree. I hit this the other day so this is why I thought it would be worth sharing it here.
without it, the system would still boot, but ostree admin status would fail to find /boot info
Well, that is up to whoever wants to carry it. I'm not very interested in that kind of thing though.
what about adding debugfs/sysfs counters on those errors?
Well, that is up to whoever wants to carry it. I'm not very interested in that kind of thing though.
what about adding debugfs/sysfs counters on those errors?
I'll propose something upstream (unless someone beats me to it)
@cgwalters are there any performance tests being run comparing ostree vs ostree+composefs+fsverity?
I am simply running:
FIO: https://fio.readthedocs.io/en/latest/fio_doc.html
fio --name=test --readonly --filename=/usr/bin/openssl --size=1M
on:
root@intel-corei7-64:/var/rootdirs/home/fio# findmnt /
TARGET SOURCE FSTYPE OPTIONS
/ /dev/disk/by-label/otaroot[/ostree/deploy/lmp/deploy/a79146c3c72ad3cf8a652a711c0d705f68a49e81eb2b0db52c079d5ce3577751.0] ext4 rw,relatime
resulting in:
Run status group 0 (all jobs):
READ: bw=229MiB/s (240MB/s), 229MiB/s-229MiB/s (240MB/s-240MB/s), io=936KiB (958kB), run=4-4msec
and on
root@intel-corei7-64:/var/rootdirs/home/fio# findmnt /
TARGET SOURCE FSTYPE OPTIONS
/ none overlay ro,relatime,lowerdir+=/run/ostree/.private/cfsroot-lower,datadir+=/sysroot/ostree/repo/objects,redirect_dir=on,metacopy=on,verity=require
resulting in
Run status group 0 (all jobs):
READ: bw=41.5MiB/s (43.6MB/s), 41.5MiB/s-41.5MiB/s (43.6MB/s-43.6MB/s), io=936KiB (958kB), run=22-22msec
I still need to check what this test is doing but I was wondering if there are any performance tests that I could use before we start deploying to embedded devices.
[apologies for the removal of previous threads but I am just learning about the tool ...so was commenting in case anyone could steer/chip in]
With something like this (10 minutes random read buffered workload) I persistently measure ~4% performance degradation on CFS reads. Does this seem correct?
#!/bin/bash
ext4="/sysroot/ostree/deploy/lmp/deploy/34da1439c2e5a9f6d57b5b48b827763f9d3d48b9f6cecd093669f61aa99b803c.0/usr/bin/"
cfsf="/usr/bin"
echo "CFS: "
cfsf_val=`sudo fio --opendir=$cfsf --direct=0 --rw=randread --bs=4k --ioengine=sync --iodepth=256 --runtime=600 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --readonly | grep READ`
echo " ==> $cfsf_val"
echo""
echo "EXT: "
ext4_val=`sudo fio --opendir=$ext4 --direct=0 --rw=randread --bs=4k --ioengine=sync --iodepth=256 --runtime=600 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --readonly | grep READ`
echo " ==> $ext4_val"
CFS:
==> READ: bw=359MiB/s (376MB/s), 359MiB/s-359MiB/s (376MB/s-376MB/s), io=210GiB (226GB), run=600001-600001msec
EXT:
==> READ: bw=371MiB/s (389MB/s), 371MiB/s-371MiB/s (389MB/s-389MB/s), io=218GiB (234GB), run=600001-600001msec
@ldts I haven't measured or anything but a 4% degradation seems reasonable with fs-verity on as it has to verify bytes as it reads.
What would be even more interesting would be, ext4 vs composefs with fs-verity on vs composefs with fs-verity off.
I expect some Desktop users would prefer fs-verity off for example so they can make some local atomic changes and maybe care less about local signatures (I dunno up for debate) and maybe don't care about fs-verity.
But in IoT or Automotive or somewhere like that it would make more sense to have fs-verity on.
With fs-verity off, I would expect composefs to be faster than ext4 as it is erofs backed, so it would be interesting to see that.
Maybe these things belong here also:
@ericcurtin ok I'll measure fs-verity off as well. makes sense. thanks for the info.!
With fs-verity off, I would expect composefs to be faster than ext4 as it is erofs backed, so it would be interesting to see that.
Note that EROFS can have some impacts on metadata access only since ostree keeps data in the underlay filesystem. If your fio workload mainly measures full-data rand/seq read access it will have minor impacts tho.
@ericcurtin ok I'll measure fs-verity off as well. makes sense. thanks for the info.!
sorry I mispoke earlier about the 4% loss (my bad, wasnt measuring the right filesystem since fsverity was enabled everywhere);
I am still doing some benchmarking (just qemu x86_64 based) but what I see with randomized reads using a buffered syncrhonous API for the tests on a full system install:
kernel:
Linux intel-corei7-64 6.6.25-lmp-standard #1 SMP PREEMPT_DYNAMIC Thu Apr 4 18:23:07 UTC 2024 x86_64 GNU/Linux
command uses fio 3.30:
fio
--opendir=/usr/bin # directory to use for reads
--direct=0/1 # buffered/direct file access (0/1): test both
--rw=randread
--bs=4k
--ioengine=sync/libaio # sync/asynch api : test both
--iodepth=256 # number of i/o units in flight against the file
--runtime=3600 # run for an hour
--numjobs=4 # four threads
--time_based
--group_reporting # report the group instead of individual threads
--name=iops-test-job
--readonly # do not write
--aux-path=~/ # use the home directory for temp files
Test
**iops-test-job: (g=0): **rw=randread**, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, **ioengine=sync**, iodepth=256**
- composefs + signed image + full integrity on rootfs: rw bandwidth ~300MB/sec
TARGET SOURCE FSTYPE OPTIONS
/ overlay overlay ro,relatime,lowerdir=/run/ostree/.private/cfsroot-lower::/sysroot/ostree/repo/objects,redirect_dir=on,metacopy=on,verity=require
- composefs : ~720MB/s
TARGET SOURCE FSTYPE OPTIONS
/ overlay overlay ro,relatime,lowerdir=/run/ostree/.private/cfsroot-lower::/sysroot/ostree/repo/objects,redirect_dir=on,metacopy=on
- no composefs**: ~670MB/sec
TARGET SOURCE FSTYPE OPTIONS
/ /dev/disk/by-label/otaroot[/ostree/deploy/lmp/deploy/d8d86dea56476c576f743097ce3a9dbee1927116893596725b39ac4bf5b17bdb.0] ext4 rw,relatime
I will stop here and comment further once all the benchmarking is done (I felt it was best if I corrected my earlier comment)
um, while testing I noticed that I cant enable image signatures without also enabling fsverity on all the files in the rootfs. Is this expected?
@ldts makes sense to me, fs-verity is what checks the signatures. Doesn't seem very useful to have signatures without fs-verity.
@ldts makes sense to me, fs-verity is what checks the signatures. Doesn't seem very useful to have signatures without fs-verity.
Arent we talking about two different things? the fs-verity kernel layer just checks any file measurements if enabled -and supported- in the filesystem.
However verifying an async ed25519 signature is a different unrelated thing. I am wondering why are they linked together,
In my use case - userspace signature validation- the public key is not loaded in the kernel key ring but just used by ostree-prepare-root to validate the composefs image signature. Other than that, I dont see why fs-verity needs to depend on it?
to me it seems like a bug against usespace signature validation- an assumption that the kernel keyring must contain the key.
said differently, why composefs image signature with fs-verity rootfs integrity is not supported - is it intentional or an implementation issue?
The current release only supports composefs image signature with fs-verity rootfs integrity&authentication
I see what you mean now, yeah I guess both could be supported individually.
also, full filesystem integrity/authentication is amazing, no doubts about it. But perhaps many embedded devices wont be able to afford it - the performance drop in read-bandwidth testing can be too noticeable. So (an extension/feature?) maybe fs-verity could be allowed to be enabled on some of the deploy folders instead of requiring it on all of them? which is what I was trying to test when I noticed it wouldn't work.
[I havent tested this patch yet, maybe it makes a good enough difference]
https://patches.linaro.org/project/linux-crypto/patch/20240507002343.239552-7-ebiggers@kernel.org/
Incidentanlly on imx8mp we are seeing a 13% improvement in read bandwidth performance tests by using ostree with CFS (without fs-verity) over EXT4. So really neat.
I feel I am polluting this thread - maybe I should open a performance evaluation issue?