Project Sparrow: KataOS

Sparrow is a project to build a low-power secure embedded platform for Ambient ML applications. The target platform leverages RISC-V and OpenTitan.

The Sparrow software includes a home-grown operating system named KataOS, that runs on top of seL4 and (ignoring the seL4 kernel) is written almost entirely in Rust.

This is a CAmkES project that assembles the entire KataOS. It exists outside the seL4 source trees since it contains code not intended to go to upstream seL4.

This uses the standard CAmkES build system by symlinking CMakeLists.txt and assumes the CAmkES dependencies are already installed. It also symlinks settings.cmake, and so retains the notion of "apps," which enables the build system to switch which assembly it builds using the CAMKES_APP CMake cache value. KataOS just has one app, system.

Sparrow Rust crates (what's included here).

[More software will be published as we deem it ready for sharing until eventually all of Sparrow (software and hardware designs) will be available.]

Many KataOS Rust crates are in the apps/system/components directory. Common/shared code is in kata-os-common:

allocator: a heap allocator built on the linked-list-allocator crate
camkes: support for writing CAmkES components in Rust
capdl: support for reading the capDL specification generated by capDL-tool
copyregion: a helper for temporarily mapping physical pages into a thread's VSpace
cspace-slot: an RAII helper for the slot-allocator
logger: seL4 integration with the Rust logger crate
model: support for processing capDL; used by the kata-os-rootserver
panic: an seL4-specific panic handler
sel4-config: build glue for seL4 kernel configuration
sel4-sys: seL4 system interfaces & glue
slot-allocator: an allocator for slots in the top-level CNode

KataOS system services make up the remaining components:

DebugConsole: a command line interface intended for debug builds
MailboxDriver: a driver for the Mailbox interface used to communicate betwen the security and management cores
MemoryManager: the memory / object manager service that supports dynamic memory management
MlCoordinator: the service that manages the vector core (using the Vector Core driver)
OpenTitanUARTDriver: a driver for the UART on the management core
ProcessManager: the service that creates & manages execution of applications
SDKRuntime: the service that handles application runtime requests
SecurityCoordinator: the service that provides an interface to the security core (using the MailboxDriver)
TimerService: a service built on top of the management core timer hardware

At the moment all the above services are included in both debug and release builds. Production builds replace the DebugConsole with a more limited interface and drop the UART driver when there is no serial console. The SDKRuntime is more a proof of concept than anything else. There are test applications written in C and Rust in the apps tree that exercise the experimental api. Production systems are likely to have their own SDK and associated runtime tailored to their needs.

Depending on Rust crates

To use crates from Sparrow you can reference them from a local repository or directly from GitHub using git; e.g. in a Config.toml:

kata-os-common = { path = "../system/components/kata-os-common" }
kata-os-common = { git = "https://github.com/AmbiML/sparrow-kata" }

NB: the git usage depends on cargo's support for searching for a crate named "kata-os-common" in the kata repo. When using a git dependency a git tag can be used to lock the crate version.

Note that many Sparrow crates need the seL4 kernel configuration (e.g. to know whether MCS is configured). This is handled by the kata-os-common/sel4-config crate that is used by a build.rs to import kernel configuration parameters as Cargo features. In a Cargo.toml create a features manifest with the kernel parameters you need e.g.

[features]
default = []
# Used by sel4-config to extract kernel config
CONFIG_PRINTING = []

then specify build-dependencies:

[build-dependencies]
# build.rs depends on SEL4_OUT_DIR = "${ROOTDIR}/out/kata/kernel"
sel4-config = { path = "../../kata/apps/system/components/kata-os-common/src/sel4-config" }

and use a build.rs that includes at least:

extern crate sel4_config;
use std::env;

fn main() {
    // If SEL4_OUT_DIR is not set we expect the kernel build at a fixed
    // location relative to the ROOTDIR env variable.
    println!("SEL4_OUT_DIR {:?}", env::var("SEL4_OUT_DIR"));
    let sel4_out_dir = env::var("SEL4_OUT_DIR")
        .unwrap_or_else(|_| format!("{}/out/kata/kernel", env::var("ROOTDIR").unwrap()));
    println!("sel4_out_dir {}", sel4_out_dir);

    // Dredge seL4 kernel config for settings we need as features to generate
    // correct code: e.g. CONFIG_KERNEL_MCS enables MCS support which changes
    // the system call numbering.
    let features = sel4_config::get_sel4_features(&sel4_out_dir);
    println!("features={:?}", features);
    for feature in features {
        println!("cargo:rustc-cfg=feature=\"{}\"", feature);
    }
}

Note how build.rs expects an SEL4_OUT_DIR environment variable that has the path to the top of the kernel build area. The build-sparrow.sh script sets this for you but, for example, if you choose to run ninja directly you will need it set in your environment.

Similar to SEL4_OUT_DIR the kata-os-common/src/sel4-sys crate that has the seL4 system call wrappers for Rust programs requires an SEL4_DIR envronment variable that has the path to the top of the kernel sources. This also is set by build-sparrow.sh.

Memory Footprint

The initial Sparrow target platform was intended to have 4MiB of memory. A production build of the included services fit in <3MiB of memory but due to the overhead of CAmkES and the rootserver boostrap mechanism actually require ~2x that to reach a running state. The kmem.sh script can be used to analyze memory use. These services should easily fit in <1MiB but due to CAmkES overhead (e.g. per-thread cost) are significantly bloated. We use kmem and the bloaty tool to evaluate memory use. Reducing memory use to <1MiB likely requires replacing CAmkES by a native Rust framework like embassy.

We also expect that replacing CAmkES C-based templates by native Rust code will significantly reduce memory use (as well as improve robustness by extending the scope of the borrow checker). The RPC mechanism used for communication between applications and the SDKRuntime is a prototype of a native Rust implementation that demonstrates this.

Portability

There are three areas in the software where per-architecture support is required:

sel4-sys: system call wrappers
kata-os-model: capDL support for the kata-os-rootserver
kata-proc-manager/sel4bundle: application construction

The first two have approximately the same architectue coverage as (upstream) seL4; there are likely issues in kata-os-model given the rootserver has never been run on any architectures but riscv32imac and aarch64. The sel4bundle supports only the riscv32 architecture used by Sparrow and in some areas is very simple (e.g. it only sets up page tables for a minimal VSpace). The goal was to unify kata-os-model and sel4bundle such that architectural support could be directly shared.

Source Code Headers

Every file containing source code includes copyright and license information. For dependent / non-Google code these are inherited from the upstream repositories. If there are Google modifications you may find the Google Apache license found below.

Apache header:

Copyright 2022 Google LLC

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

aegistudio / sparrow-kata-full