Exploring potential collaboration between Kairos and Bootc projects

[Ettore Di Giacinto]

hello! 👋

I've just learned today about https://github.com/containers/bootc ! I tried to reach over matrix but was redirected here - I'm the maintainer of https://github.com/kairos-io/kairos which is a project that have many crossing points with bootc, and I was wondering myself if you would be interested in having a chat! I see there are community events for podman but none for bootc. Just curious, how does it handles upgrades without ostree? is it a/b style? is the OS immutable or takes over only the upgrade process?

Best

[Colin Walters]

Hi, thanks so much for filing this! At a quick glance, I'd say there's indeed 85% overlap between the goals and methodology of the projects. But there's a lot of interesting stuff in that remaining 15% 😄

First and most obviously, bootc is a brand new project written mostly by one person with little documentation and no production use. Whereas it looks like Kairos is much more mature. But the point is that the current backing code of bootc (mostly ostree and the new ostree Rust) is also very mature with lots of production use - this is just a thin new "frontend" to it. (There's literally just ~500 lines of code in this repo today, though that will change once #30 lands).

I am totally open to collaboration! I think once bootc progresses a little bit we will start having community meetings indeed, or to piggy-back on the podman ones for 5-10 minutes perhaps.

I'd also be open to joining Kairos community events. To start, I'm now in the Matrix channel.

Offhand, some things in that remaining 15%:

• A major point of bootc is that making custom images is literally the same process as application images. What you push to a registry is exactly the same filesystem as what's pulled and booted, whereas it looks to me like Kairos is partition-oriented, so has an extra step where an ISO is created and pushed to a registry? I need to dig into this a bit more.
• This isn't stated directly but for me bootc has a toplevel goal to enable seamless in-place updates for existing ostree users. Because we have a lot of production users we can't just drop or even require to reprovision.
• bootc aims to support all Linux types, not just Kubernetes nodes. Specifically including "standalone" and non-cloud systems, like IoT and desktops. This leads into the next bit:
• The details around partition setup, specifically things like /etc being ephemeral clash with the above. On a desktop, I don't want my WiFi configuration stored in /etc/NetworkManager to be flushed on reboots. Similarly for bare metal servers, static IP addresses are a real use case. On a cloud, sometimes I do want /etc flushed, sometimes I don't.
• bootc (and ostree) have first-class support for SELinux. This affects a rather surprisingly large set of design choices in implementation.
• bootc is not attempting to do things like build cloud images and ISO images itself; I think we should leave that to distribution tools. Or at least, maybe this project would have "demonstrators" of that.
• We've been making an increasing investment in Rust for the last few years from both teams that are involved in this project (coreos/ team and the containers/ team). Obviously Go will be here for the remainder of my career at least, but I've been repeatedly burned by its "superficial simplicity" and using Rust for bootc was a very deliberate choice.

Just curious, how does it handles upgrades without ostree? is it a/b style?

It doesn't at all today, but indeed part of the idea is that we should also support that flow. I'd like to investigate standardizing or at least collaborating on shared code for things like how to store and manage raw block devices stored in a registry. The obvious thing of just sticking a disk image in a tar layer is painful from a bandwidth perspective; in coreos we have tooling called "osmet" that synthesizes disk images from ostree and tar (much like jigdo).

Thanks again for filing this and bringing Kairos to my awareness. Looking forward to continuing this conversation and collaborating!

[Ettore Di Giacinto]

Thanks for the quick answer!

Hi, thanks so much for filing this! At a quick glance, I'd say there's indeed 85% overlap between the goals and methodology of the projects. But there's a lot of interesting stuff in that remaining 15% smile

First and most obviously, bootc is a brand new project written mostly by one person with little documentation and no production use. Whereas it looks like Kairos is much more mature. But the point is that the current backing code of bootc (mostly ostree and the new ostree Rust) is also very mature with lots of production use - this is just a thin new "frontend" to it. (There's literally just ~500 lines of code in this repo today, though that will change once #30 lands).

I am totally open to collaboration! I think once bootc progresses a little bit we will start having community meetings indeed, or to piggy-back on the podman ones for 5-10 minutes perhaps.

Awesome :) looking forward to hear more

I'd also be open to joining Kairos community events. To start, I'm now in the Matrix channel.

Offhand, some things in that remaining 15%:

* A major point of bootc is that making custom images is _literally the same process_ as application images.  What you push to a registry is _exactly the same filesystem_ as what's pulled and booted, whereas it looks to me like Kairos is partition-oriented, so has an extra step where an ISO is created and pushed to a registry?  I need to dig into this a bit more.

That's the same approach of Kairos. The image being pulled can be run with docker, podman, etc as a standard application, but still bootable from Kairos without any container engine (it does loopback mount the image behind the scene). As per partitioning, it is completely flat, and used just to store bootable images: https://kairos.io/docs/architecture/immutable/#design

Additional tooling is in place to convert a container image to a bootable ISO, or a raw disk.

* This isn't stated directly but for me bootc has a toplevel goal to enable seamless in-place updates for existing ostree users.  Because we have a lot of production users we can't just drop or even require to reprovision.

I see! that's interesting and could be a meeting point. I always wanted to explore an ostree backend for Kairos, and maybe bootc might be at hand exactly for that.

* bootc aims to support _all_ Linux types, not just Kubernetes nodes.  Specifically including "standalone" and non-cloud systems, like IoT and desktops.  This leads into the next bit:

* The details around partition setup, specifically things like `/etc` being ephemeral clash with the above.  On a desktop, I don't want my WiFi configuration stored in `/etc/NetworkManager` to be flushed on reboots.  Similarly for bare metal servers, static IP addresses are a real use case.  On a cloud, sometimes I do want `/etc` flushed, sometimes I don't.

Right, this indeed isn't really obvious from our docs but Kairos while has primarly focus for Kubernetes it does handle well also outside of it. /etc is ephemeral, but partially as it can be completely customized to be flushed or not. Indeed for Kubernetes we need several paths to be writable and persistent into /etc, as well as some systemd related ones.

* bootc (and ostree) have first-class support for SELinux.  This affects a rather surprisingly large set of design choices in implementation.

Altough we don't support it directly yet, it is in our roadmap and I do agree it tends to be a can of worm, but I see a lot of crossing points between our projects!

* bootc is _not_ attempting to do things like build cloud images and ISO images itself; I think we should leave that to distribution tools.  Or at least, maybe this project would have "demonstrators" of that.

👍

* We've been making an increasing investment in Rust for the last few years from both teams that are involved in this project (coreos/ team and the containers/ team).  Obviously Go will be here for the remainder of my career at least, but I've been repeatedly burned by its "superficial simplicity" and using Rust for bootc was a very deliberate choice.

👍 I don't have a strong opinion here, but I welcome Rust. I think languages are just barriers to what someone wants to really achieve and the computer, so I don't mind, and there are obviously better choices than others based on the scope of the software.

Just curious, how does it handles upgrades without ostree? is it a/b style?

It doesn't at all today, but indeed part of the idea is that we should also support that flow. I'd like to investigate standardizing or at least collaborating on shared code for things like how to store and manage raw block devices stored in a registry. The obvious thing of just sticking a disk image in a tar layer is painful from a bandwidth perspective; in coreos we have tooling called "osmet" that synthesizes disk images from ostree and tar (much like jigdo).

For kairos we don't store separate disks and push them as containers images - we flatten the container image in runtime when we write it to disk so there is still one source of truth - you can run the container locally as an app, or just boot it seamlessly.

Thanks again for filing this and bringing Kairos to my awareness. Looking forward to continuing this conversation and collaborating!

Thanks to you! It was good to understand to set expectations and understand where bootc is heading to, I'll be joining community events as soon as they are up!