potano / layercake

Layercake build-root manager

Layercake

Gentoo’s Binary package guide describes how to set up a host for building binary packages to be deployed on one or more target systems. This allows the administrator to build once and deploy multiple times. An added benefit is that the target machines remain fully usable during the build process.

A major limitation of the methods presented in the Binary-package guide is that the target machines must have the same CHOST, CFLAGS, CXXFLAGS, and USE-flag settings as the host which builds the packages. If the processors on the various machines are of different generations, the processor-specific USE flags must follow the lowest common denominator.

Gentoo users are already familiar with using a chroot in the process of setting up a Gentoo Linux system. This ensures that the setup is done cleanly from the initial stage tarball. In the same way, using a chroot on the binary package host lets the user target a configuration which varies significantly from that of the host system. It also opens the possibility of having multiple build roots on the same build host, each with its own target configuration.

Layercake makes this easy to manage:

• Target systems may be configured differently than the build host.

• Host systems need not use Gentoo Linux; most other Linux distributions should work.

• A build host may contain multiple build roots, each with its own USE-flag and other settings.

• A single system-wide Portage tree (plus overlays) may be shared among all build roots; no need to sync individual trees.

• Supports similar system-wide sharing of other directories such as downloaded tarballs (distfiles directory)

• Maintains symlinks needed for a web server to access binary-package directories. Also maintains such symlinks for any supplemental-file directories the user may configure.

• Automatically performs any needed mounts, including /dev, /proc, /sys, and shared directories as configured. Each build root has its own mount configuration.

• Places each build root with its associated files and directories into its own containing directory.

• Includes a separate executable, stagemaker, that generates customized stage tarballs for setting up new target machines. Such tarballs, when stored in the supplemental-file directories mentioned above, become available for download from the build server.

• Sets the PS1 environment variable at chroot time to indicate not only that a chroot is active but also the name of the build root.

• Allows multiple simultaneous chroot sessions on a build root. Layercake does mount operations only when needed and checks that any existing mounts have the correct endpoints. (It is still up to the user to ensure that no more than one emerge operation is in progress at a time.)

Layercake has a major feature which can result in substantial savings in build time and disk space. In cases where two or more target systems have mostly the same USE-flag configuration and similar architectures, generally only a few packages will have actual USE-flag settings which differ between targets. That means that from one build root to another most packages turn out to have the same configuration. By means of OverlayFS mounts, Layercake can mount a build root on top of another. When the user builds or updates packages in the lower build root, the installed packages are visible in the upper build root—​and a world update in the upper layer would have to build only those packages with different USE flags. This makes it so that there is much less work to do in the upper build root—​and a reduced space requirement in the build host.

This support for overlay mounts of one build root on top of another gives rise to the name Layercake. The layers of this layercake are the build roots plus their associated data. At least one layer must be a base layer, meaning that it is built up from a stage file in a process that is largely the same as described in the Gentoo Handbook. One or more derived layers may be layered on top of a base layer or even another derived layer. Layercake places no limit on the height of such a stack of derived layers. Since kernel version 4.0 or so, neither does OverlayFS.

A layer directory is a directory that holds all the filesystem entries for a layer: the build root, a configuration file, the OverlayFS directories for the build root, and so on. A layer’s name is the name of its layer directory.

Note	Overlay mounts work only if the Overlay Filesystem (CONFIG_OVERLAY_FS) is enabled in the running kernel.

Note	Layercake currently has no support for cross compilation. All build-time dependencies must be able to execute natively on the host machine. This means that the CHOST and -march settings in all layers must be compatable with the host hardware.

Layercake and Stagemaker are written in the Go language and have no dependencies beyond the Go standard library. They build to statically linked executables that include defaults that permit operation without configuration files. Installation from source is almost trivially easy.

Basic Usage Patterns

These steps assume the user has configured and initialized the Layercake installation in an operational Gentoo host environment. Many configurations are possible. For details about the setup, see Setting up the Layercake environment.

These examples assume further that the build host has the name "builder" and that we have set up a web server that listens on port 1000 to serve documents from the Layercake exports directory.

Create, populate, and use a base layer

We use the term base layer to denote a layer built up fully from a stage tarball according to instructions in the Gentoo Handbook.

In this example we use "rosebud" as the name of the layer and intend to build the layer with a USE-flag setup to match an already-deployed target. Start with this command:

layercake add rosebud

This generates the layer along with its build-root directory and layerconfig file. Note the basic calling form of Layercake: the first positional paramter is the name of a command; additional positional parameters are arguments.

Whereas the Handbook has you create, mount, and chdir to a new root filesystem, use this command instead:

layercake shell rosebud

This opens a shell in the build root directory. Here is where you select, download, and install a stage tarball and set things up in in preparation for the chroot. You can save time by copying Portage files from your existing target. Don’t forget to copy /etc/resolv.conf from the host system.

In most cases, there is no need to edit the layer’s configuration file. If there is, it is easily reached at this point because it is in the parent directory of the build root.

When setting up make.conf, be sure to add the buildpkg and binpkg-multi-instance features to the FEATURES variable. (You may omit binpkg-multi-instance if you will never use OverlayFS mounts.)

Once the build root is set up with the basic configuration, exit the current shell and enter the chroot with this command:

layercake chroot rosebud

If you get a message about a mountpoint directory not existing, fix the problem and try the layercake chroot again.

Of note: for these examples, we assume we are using the host system’s Portage tree and overlays at the directory /var/db/repos. Since this is the default configuration and the layercake chroot command makes the mounts automatically, there is no need to download the initial Portage tree or to do emerge sync within the chroot.

As to steps the Gentoo Handbook gives at this point

• Skip mounting the boot partition

• Skip emerge-webrsync and emerge --sync unless running on a non-Gentoo build host

• Select the desired profile, which might well be a custom profile

• Set up the USE variables

• Set up the time zone if desired

• Set up the locale

• Skip setting up the kernel and initramfs but emerge firmware if the targets will need it

• Skip setup steps for network, fstab, root password, and bootloader

The layer is fully accessible from other Bash prompts. There is no problem at all for a user to issue a layercake chroot command to have an additional chroot into the layer or to issue layercake shell to switch to the layer’s build root while still keeping access to the full host filesystem. The latter can be quite helpful if your editor of choice is not nano.

Build all the packages your target will need. Once done, you may exit the chroot. Note that this action never unmounts anything. To unmount a layer (short of rebooting or unmounting by hand!) issue the layercake umount command:

layercake umount rosebud

Note that it is rarely necessary to unmount a layer.

On the target host

Assuming that we have set up the layer with USE flags and march to match an existing target system and then built a full set of packages for it, we can set up the target to use the build root by setting the following entries in the target’s make.conf:

• FEATURES="getbinpkg"

• PORTAGE_BINHOST="http://builder:1000/packages/rosebud"

Now an emerge world should pick up all its new packages from the build host.

Create and use a child (derived) layer

This is much simpler than setting up a base layer. Here we assume the derived layer, called "notebook", has a different set of processor flags than does rosebud.

layercake add notebook rosebud

This form of the add command specifies the base layer ("rosebud") for the new layer ("notebook"). Since this is not a base layer, we can go directly to the chroot:

layercake chroot notebook

Edit the Portage configuration as necessary and build according to the new configuration with the confidence that the base layer remains unaltered.

This time we need to set up a new machine. We could start with a recent stage3 file on that machine, but we would likely be faced with rebuilding packages on that machine since some of the packages in the @system set would now have different USE flags. We avoid this problem by using stagemaker to generate a stage3 tarball that matches the build environment.

Since we write the new stage tarball to a directory outside the build root (and also because stagemaker itself is not part of the build root), we cannot stay in the chroot to generate the stage. Exit the chroot and issue these commands to generate the stage tarball:

layercake shell notebook
mkdir ../generated
layercake mount notebook
stagemaker -generate -o ../generated/stage.tar.gz

The layercake shell command takes us to the build directory, the mkdir command creates the directory to hold the stage file, the layercake mount command ensures the needed symlink is visible to the web server, and the stagemaker command generates the actual stage tarball.

Note that while Stagemaker is designed to operate with an arbitrary root directory and has no requirement to be confined to a chroot, nothing keeps it from working in a chroot with the -root / option. It is a static executable with no configuration file needed.

On the target host

Now as we set up the new target, we download the stage tarball with

wget http://builder:1000/generated/notebook/stage.tar.gz
tar xzpf stage.tar.gz --xattrs-include='*.*'

or, if you are more adventurous,

curl http://builder:1000/generated/notebook/stage.tar.gz | tar xzp --xattrs-include='*.*'

The --xattrs-include switch is still required for Stagemaker-generated tarballs in order to extract file capabilities, but --numeric-owner is not because Stagemaker does not store user or group names.

Stage tarballs that come from the Gentoo upstream contain vanilla copies of all files in /bin, /etc, /usr, and /var as generated from the ebuilds in the system set plus vanilla /etc/portage settings. By contrast, the stagemaker utility copies over any edits made to these files in the build root. This is a handy way to propate settings you will use all the time, but there are caveats as regards the Portage files /etc/portage/make.conf and /var/lib/portage/world.

In make.conf on the target machine, be sure to remove buildpkg and binpkg-multi-instance from the FEATURES variable and add getbinpkg. Add a PORTAGE_BINHOST variable to point to your build host.

• FEATURES="getbinpkg"

• PORTAGE_BINHOST="http://builder:1000/packages/rosebud"

The world file in the generated stage tarball includes the full set of entries in the build root’s /var/lib/portage/world. Since you may want only a subset of these packages on the target machine, so be sure to edit out any such lines before emerging world. Edit as needed and then do the world update.

Now you may finish the setup as the Gentoo Handbook instructs.

More information

• The Layercake Manpage details Layercake command invocation.

• Layercake directory layout

• Layercake configuration

• The stagemaker utility lets you customize the stage tarball in helpful ways. You can add files such as helper scripts for setting up the bootloader or rsyncing the Portage tree from the build host. You can also add packages from the build root that are beyond the @system set, such as eix, gentoolkit, and your favorite text editor—​with proper dependency resolution. See the Stagemaker documentation for details.