Support for converting containers

[Sean McNamara]

Containers, e.g. using LXC, LXD or a Proxmox CT, do not use either UEFI or BIOS and in fact do not run their own kernel. It would be nice if Convert2RHEL could support converting a CentOS 8 container to RHEL 8.

[zhukovgreen]

@allquixotic thank you for this feature proposal. Could you explain how can I try the container conversion? What is your use case?

I was trying in the past to convert the docker container (just for curiosity) - it went well before the reboot phase:)

[Sean McNamara]

Hello,

My main use case would be the following:

1. Install Proxmox VE 6.3 on bare metal.
2. Create a new Container Context ("CT" in Proxmox terminology) for CentOS 8. These CTs are based on LXC technology. Make your CT an unprivileged container.
3. Boot the container, configure networking, and install Convert2RHEL.
4. Run convert2rhel --username ... --password ...

Expected result: successful conversion to RHEL.
Actual result: The following checks in Convert2RHEL fail in an LXC container:

• UEFI check, especially if the Proxmox VE host happened to be installed via UEFI (the path /sys/firmware/efi is readable from a container by default)
• All checks pertaining to the kernel modules
• The /sys mounted read-write check. It is generally mounted read-only in an unprivileged container, because if it were read-write, a container would be able to do something really impactful system-wide like enable/disable IPv4 forwarding, or so many other things that are not virtualized or controlled on a per-container basis when using LXC or LXD.

I have manually edited the Convert2RHEL code (in very dirty ways) to unconditionally bypass or remove the above checks. Everything seems to almost work, except that the filesystem package from RHEL8 expects to be able to write to /proc during the install. This was a documented issue somewhere else but I can't remember where it's documented now. I'm not sure of the workaround for it.

Converting a container to/from unprivileged to privileged and back is non-trivial, because an unprivileged container uses a namespace for uids and gids, so for example, from the perspective of an unprivileged container, files owned by root with a uid of 0 in the container are actually mapped to uid 100000 in the "real" filesystem data (i.e. from the perspective of the host). In a privileged container, files owned by root are given uid = 0 from the perspective of both the container and the host, so the actual file metadata contains the "unmapped" actual uids from within the guest. A privileged container is also technically allowed to do many types of things that can break isolation between container and host, especially the root account.

For that reason, converting to RHEL as a privileged container and then going back to unprivileged is not preferred. It's also a bad idea in case the Convert2RHEL script or any of the RHEL package installation scripts try to do weird things like write to /sys or load kernel modules, which might "work" but have unintended consequences in a privileged container.

Unprivileged containers are supposed to be, if not exactly as isolated as a VM, nearly so -- at least enough to prevent many/most accidental changes to the host from within the container, even as root. And I believe they have a longer-term goal of providing "multi-tenancy isolation" -- the ability to 100% safely partition out the resources of a physical machine to someone you don't trust, give them a container, and they won't be able to "break out" of that container and alter or observe the physical host or other containers/VMs.

I don't think default LXC configuration of an unprivileged container is 100% to being multi-tenant safe, but it's getting there. In the interim, OS containers on Linux (such as those implemented using LXC or LXD) are a good way for separating out different workloads for whom you trust the administrator, or you yourself are the administrator, especially if you want to use different operating system or software stacks for different purposes. They serve a similar role to VMs, but without any real overhead, while VMs have very significant and measurable overhead (especially with file and network I/O), even on the latest hardware platforms.

All of this is to say, RHEL8 in a Proxmox VE CT is highly desirable to me, because I love the performance of bare metal containers, and I love the flexibility of RHEL AppStreams + RHEL support + SCL + long-term security update lifecycle, while CentOS 8 is going away as a product in 2021. So that is why I want to run RHEL in an LXC container on Proxmox VE.

P.S. - I know that the concept of isolated "Operating System Containers" (OS Containers) is a foreign concept in Red Hat land, but the technology does exist in the Linux kernel to support them. It comes down to how robustly a tool like LXC can combine primitives like cgroups, namespaces, and Linux Security Modules (like AppArmor, which is popular with LXC due to that project's close relationship with Ubuntu) to provide the correct abstractions and isolations.

OS Containers behave like virtual machines but without the need to virtualize hardware, and I/O performance of OS Containers is equivalent to that of bare metal. For example, even the fastest hypervisors add latency to the network interface, but in an OS Container there is not any measurable increase in network latency. I do wish that Red Hat would seriously embrace an existing OS Container technology within their ecosystem, or at least work on a new technology of their own building OS Containers that are supported by Red Hat. The concept is an old one, with the first serious commercial implementation coming from Sun Microsystems in Solaris called "Zones". It always feels like a waste to me to run things in VMs, knowing just how airtight Sun was able to make Zones more than 10 years ago.

[zhukovgreen]

Thank you @allquixotic for this interesting use case and your thoughts on the benefits of the OS containers. I forwarded this information further within the Red Hat.

So far what I could say that we're working on supporting the UEFI (I see you already noticed), and very soon we'll introduce the possibility to skip certain checks (i.e. kmods, to /sys etc.).

I'll ping you when this will be ready

[Michal Bocek]

Hi @allquixotic, Red Hat does not support LXC (nor LXD, nor Proxmox): https://access.redhat.com/solutions/3786491.

Nevertheless, we'd be happy if you or the Proxmox community add support through a pull request.

[mightyspiky]

I have manually edited the Convert2RHEL code (in very dirty ways) to unconditionally bypass or remove the above checks. Everything seems to almost work, except that the filesystem package from RHEL8 expects to be able to write to /proc during the install. This was a documented issue somewhere else but I can't remember where it's documented now. I'm not sure of the workaround for it.

@allquixotic , The workaround to make filesystem package reinstall/update successful is to add this to /etc/rpm/macros.dist :

%_netsharedpath /sys:/proc

Don't know what it does exactly but it sure works. I've just converted with success a Rocky Linux 8.7 unprivileged LXC container to RHEL 8.7 after doing the above modification and after bypassing some specific checks of convert2rhel.