xc is a work-in-progress container engine for FreeBSD. This README document is intended for contributors, developers or anyone curious about this project to understand, build, run and contribute. This is not a step-by-step documentation, and such documentation (although is work-in-progress and potentially a bit outdated compare to the main branch) can be found here
xc consists of 3 binaries, xc (client), xcd (server), ocitar (OCI layer archive helper). xcd depends on ocitar to run correctly, hence ocitar should exists in under one of xcd's $PATH directories.
All three binaries can be built simply by runnig cargo build in the source directory.
- FreeBSD 13 stable or newer (tested on both arm64 and amd64)
- cargo (require nightly if intended to build USDT probes for
xcd) - cmake (sqlite build dependency)
Copy xc, ocitar, and xcd to one of the $PATH directories from build directories ($src/target/release for release build, $src/target/debug for debug build)
xc now take yaml configuration. The scheme (struct XcConfig) for the configuration file can be found at xcd/src/config/mod.rs. By default, xcd looks for the configuration at /usr/local/etc/xc.config
The core of xc is xcd, the daemon handles basically everything. xc, the client program, submit requests to the daemon via UNIX socket, typically at /var/run/xc.sock. Unlike similar container technology such as docker, xcd does not accept HTTP requests but instead accepts JSON encoded requests, sometimes with file descriptors.
Every request xcd receives contains a method name and the corresponding payload. There are macros available to generate new methods to extend the features of xc. See $src/xcd/src/ipc.rs for examples.
The macro to define a new method also creates client-side helper functions pub fn do_$method(..) and can be used in the xc client program.
The global state of the daemon is called Context, and is defined in $src/xcd/src/context.rs.
The global state (Context) owns a number of Sites. A Site is essentially an abstraction of "a place a container lives in". Think Context is a landlord, a Site is a portion of land the landlord rents out.
The purpose of this abstraction is to separate the duty of cleaning up a container. System-wise resources are made to clean up at the Site level, for example, destroying ZFS datasets, releasing IP addresses, etc, things that the tenant (container) shouldn't, and couldn't care about. This allows the global resources to always cleanup no matter what happened in the container to cause an exit (Jail cannot be created, precondition failure, executable crashed, cannot run the executable, etc...).
This is also planned to support FreeBSD containers that require multiple hosts to function in the future. More specifically, Root-on-NFS Jails, whose root filesystem may be exported by a different host than the host running the processes. In these cases, each host owns a site that references/relates to one container.
On the other hand. Once a site is created, the daemon process fork and run a kqueue backed run loop in the child process. This run-loop is responsible for spawning and reaping processes in the Jail, as well as collecting matrices. The site communicates with the run-loop via a UNIX socket pair, which sometimes also forwards file descriptors received from xc client to the run-loop. For example, in the case of xc exec without pty, the stdout and stderr file descriptors of the xc client process are first sent to the xcd daemon, which is later forwarded to the run-loop to use as the stdout and stderr of the new process.
Reaping is done by tracing the PIDs via NOTE_TRACK of EVFILT_PROC of kqueue. This allows us to reap processes without having an init in the Jail nor using procctl. The benefit of not using an init is to allow us only to track selections of process sub-trees that are directly related to the container lifetime. By doing this, we can prevent some long-running processes irrelevant to the container's lifetime (such as profiling/analytics) from stopping the container from exiting.
By default, unlike in Docker, xc waits for all descendants of the main process to exit before killing the container, instead of just the main process. In other words, processes such as nginx that immediately daemonize itself can run un-modified without special flags or init.