GNU/Linux on Apple M1-based hardware, including, in particular, a MacBook Pro called “pearl”. No relationship to Perl, though we do use that.
Ideally, this repository will contain everything you need to build and run Debian GNU/Linux on any M1 machine.
We distribute two Mach-O files, which can be installed using kmutil on an M1-based MacOS machine and will start a usable Linux kernel, in addition to being able to do other things. The small distribution assumes that a Linux root file system has already been set up, the large distribution doesn’t and includes a minimal Debian root file system.
The kernel is based on the work done by Corellium, but will be updated to include any drivers upstreamed by the Asahi Linux project, and hopefully to include further drivers and features absent from the other two projects.
It would be much nicer to have one project rather than three of them, but there appears to have been a falling out between contributors to the other two projects, and the last I’ve heard is that the Asahi Linux project’s main contributor (or benevolent dictator) has decided not to include any code from (or credit for) Corellium in the drivers to be submitted to upstream Linux. In addition, it has been said that even simply looking at those drivers and using them as a source of documentation is frowned upon.
In addition, the FDT device trees used by the Asahi Linux project are going to be very different from the one used by Corellium.
The Asahi Linux project has decided to go with a long chain of complicated and somewhat independent boot loaders as part of their boot process: it has been suggested that this sequence might look like this:
- iBoot
- m1n1
- U-Boot
- GRUB
- Linux
By contrast, the Pearl boot process looks like this:
- iBoot
- Linux
- Linux
- Linux
That means we only need to write device drivers once, and we can use all devices that have a Linux driver.
The current status is that Pearl is a usable everyday desktop system, including everything I use on a daily basis, with the exception of audio support.
| Feature | Upstream | Asahi | Corellium | Pearl |
|---|---|---|---|---|
| kernel version | 5.13.0-rc4 | ? | February 2021 | 5.13.0-rc4 |
| audio | no | no | no | no |
| bluetooth | no | no | no | no |
| resume/suspend | no | no | no | no |
| SMP | partial | partial | yes | yes |
| keyboard/touchpad | no | no | yes | yes |
| backlight blanking | no | no | no | yes |
| keyboard backlight | no | no | no | yes |
| kexecs other kernels | yes | yes | no | yes |
| KVM support | no | no | no | yes |
| WiFi | no | no | yes | yes |
| USB host support | no | yes | yes | yes |
| USB gadget support | no | no | no | yes |
| NVMe support | no | no | yes | yes |
Testers are desperately wanted! Please contact me at pipcet@gmail.com if you have M1 hardware and want to try Pearl.
This repository contains somewhat minimalistic code to turn a Linux kernel image (such as arch/arm64/boot/Image) into a Mach-O object file which can be loaded by iBoot. There is some boilerplate code because iBoot does not allow Mach-O kernels to make alignment requests, but apart from that, we jump straight into the kernel.
That initial kernel does not rely on the existence of a complete device tree; instead, it sets up a minimal device tree (but it does so in the kernel, with all library functions available and no need for duplicating them in the boot loader) which contains the ADT range, the kernel launches userspace, userspace code transforms the ADT into an FDT, and userspace finally kexecs a second kernel which makes use of the FDT to talk to hardware.
That second kernel can then perform as much or as little I/O as the user desires, using standard Linux device drivers, to load a third kernel.
For debugging and development, the second kernel can also load something different, such as m1n1 or U-Boot (further options would be MacOS or GRUB, but those don’t work yet), but that’s not what the ordinary boot process would look like.
As a special case, the second kernel can load an updated version of itself; this is important because the first version is signed by kmutil and is difficult to replace, while the second version need not be and can be provided over the net straight from a development machine.
What is installed using kmutil is a Mach-O containing:
- minimal boilerplate code to realign the image
- a Linux kernel image for the first kernel, containing:
- an initrd; containing:
- a Linux kernel image for the second kernel
- tools to manipulate the device trees
- kexec, to execute the second kernel
- [optionally] a default Linux kernel image for the third kernel
- [optionally] code for a boot menu
- [optionally] lvm2, cryptsetup, etc.
- [optionally] m1n1, as a Mach-O file
- [optionally] U-Boot, as a Linux image file
- [optionally][some day] the MacOS kernel, as a Mach-O file
- an initrd; containing:
The first kernel does not have device drivers and cannot interact with the user in any meaningful fashion; it cannot even reboot the machine.
The second kernel does have device drivers, so it can be controlled using the keyboard; it also presents a USB gadget on one of the USB Type C ports which can be used to control it or upload a third-stage kernel. Unfortunately, I do not have hardware with fixed Ethernet connectivity, but it would certainly make sense to boot over the network. Booting over WiFi should also be supported at some point, though it requires additional blobs.
There is a single compressed initramfs included in the Mach-O image. The second kernel is launched with a second initramfs which is constructed by the first kernel, and mostly identical to the first initramfs. The third kernel is also used with an initramfs, which is provided along with that kernel.
Unfortunately, the WiFi module requires firmware which currently needs to be copied from the MacOS installation. For legal reasons, we can’t provide it.
Similarly, the MacOS kernel itself cannot be redistributed.
Furthermore, WiFi requires knowing the ESSID and passphrase for a network, and asking for that on every boot is annoying.
So I’m working on a facility to include an arbitrary “blobs” tarball containing those files, to be created by the user as part of the build process.
The protocol used to start secondary CPU cores on M1 machines cannot be restarted in the obvious fashion: once the CPU is running, it can never go back to behaving as it did initially.
It’s possible to virtualize this, or otherwise expose a mechanism for a started CPU to go back to a spin table whence it can jump into another kernel, but that requires code duplication.
Instead, Pearl simply leaves the starting of the secondary CPUs to the final kernel, leaving them in the same state as iBoot. That means less performance for the first stages, but that’s not expected to become a problem.
The Pearl images do not use m1n1 or U-Boot as part of the normal boot process. There are many reasons for this, but the main reason is that the historic reasons for the existence of boot loaders do not apply to the M1 platform.
No. It doesn’t. Linux should accept hardware in whatever reasonable state it’s in, and the boot loader should be free to leave hardware in any reasonable state.
No. It’s a non-trivial task, requiring code which pulls in many dependencies (to do it properly, at least). It’s best left to userspace, and it’s entirely possible to launch the initial userspace without an FDT.
The minimal device tree set up by the kernel itself is based on the boot args structure which does specify the frame buffer parameters.
It’s true that they are larger than they would otherwise be, but I don’t think that’s relevant at this point. A complete image will be somewhere in the 30-40 MB range, still much less than MacOS. Images which contain Debian root file systems are obviously larger.
There is no code that would need to. All we do is realignment (and that’s an unfortunate iBoot limitation), then we’re in the kernel image and use whatever printing functions are enabled there.
I consider that a good thing.
I don’t understand this point at all. Changing a kernel should be trivial, it’s changing the boot loader that is cumbersome and hard to do, requiring physical interaction with the MacOS Recovery Mode. We should minimize having to do that.
I see absolutely no reason to cooperate with the Apple boot process more than necessary. The right approach here is to install a single Mach-O “kernel” image, then never touch it again. Long-pressing the power button is annoying and unnecessary; it should not be required for ordinary day-to-day use of MacOS and Linux.
We do, which is why you can kexec m1n1.
Works, but appears to have been abandoned for now.
Still in the very early stages.
The U-Boot port at https://github.com/kettenis/u-boot
This works well enough to load OpenBSD from disk, apparently!
The OpenBSD port at https://github.com/openbsd/src
Feel free to contact me at pipcet@gmail.com, on GitHub, or in any of the other usual ways. So far, there has been too little communication rather than too much of it.
Both Linux FDT device trees and Apple ADT device trees represent hierarchies of nodes containing other nodes and leaf properties; each property has a name and contains a sized array of untyped data.
I’ve found it convenient to represent them in a simple text format containing lines like this:
top.middle.lower.property = <0x12345678>
It’s much easier to manipulate such lines using standard GNU/Linux utilities.
We need to parse, without including too many dependencies, various binary data:
- Mach-O images
- the bootargs structure
- ADT
- FDT
- Linux image files
I’ve decided on using a simple ad-hoc perl solution for that, rather than including Python in the initramfs. It is, however, a little nicer than the built-in unpack function: ADTs are represented as:
sub adtnode() {
struct [
count(props => u32),
count(nodes => u32),
props => repeat(\&adtprop),
nodes => repeat(\&adtnode),
]
}
sub adtprop() {
struct [
name => string 32,
size => size(data => u32),
data => data,
align(4)
]
}I’m trying to compress everything once only, but currently the initial kernel image is actually uncompressed; the initramfs is compressed, though.
The second stage presents a USB gadget exposing an ACM interface (which is piped to a shell) and a mass storage interface (which can be used to write an image to be unpacked and executed). There’s also an Ethernet device, but that’s not used yet.
Piped directly to a shell.
Quite simple, but sophisticated enough to prevent writing to random USB devices, and also to ensure writes are not reordered to the point where we try to boot a partially-loaded kernel.
The commfile is currently limited to 1 GB.
We’re using kexec-tools without any additional modifications, but the kernel includes patches by @mzyngier to properly exclude reserved memory ranges as possible locations for kernel images.
We’re reusing some CI/CD scripts I’ve written for other occasions, so artifacts (one including Debian, one not including it) are produced automatically by pushes to the main branch, and releases containing those files are produced automatically by pushes to the release branch. That means that the precise files included in the automatic releases haven’t been tested, and often the similar artifact versions haven’t been tested, either.
There are a number of ADT properties representing, in one of several ad-hoc formats, “tunables”, which describe bits to be set and cleared in MMIO space. The Corellium pre-loader code translates those to a common format to be applied by Linux drivers, increasing the number of ad-hoc tunable representations by one.
We do the same thing, but we use userspace code rather than pre-loader code.
One particular issue is the existence of fuse maps which represent values to be copied from one register to another. What the Corellium code does is to read the source registers at pre-loader time; our current code reads them from userspace using /dev/mem. Ideally, we’d read them only when they’re actually applied, but that would necessitate yet another ad-hoc format to encode them.
Is a direct boot supported from <row> to <column>?
| iBoot | stage2 | m1n1 | U-Boot | GRUB | Linux | |
| iBoot | no | yes | yes | no | no | yes |
| stage2 | no | yes | yes | yes | no | yes |
| m1n1 | no | yes | yes | yes | no | yes |
| U-Boot | no | yes | no | ? | yes | yes |
| GRUB | no | yes* | no | no | no | yes* |
| Linux | no | no | no | no | no | no |
(* - requires EFI_STUB build, untested)
The main problem, right now, is that USB ports cannot be switched freely between host, gadget, and power modes. Right now, it is fixed that the first port is always in power mode, the second port is in gadget mode during the second stage and in host mode otherwise.
This will need to be changed to support Macs with USB keyboards, though those tend to have enough USB ports that this shouldn’t be a problem.
MacOS appears to have a bug which makes it reset the backlight level to “very dim” when it is booted. However, it’s possible to use nvram to set the right level in Recovery Mode.
It’s possible to disable the somewhat annoying boot chime using nvram or the MacOS configuration utilities.
We’re currently using 16 KB pages, while most distribution kernels use 4 KB pages. The CPU’s MMU supports using 4 KB pages, but it appears the IOMMU does not, and the Linux code assumes identical page sizes for both.
This breaks Debian’s Emacs package, but the bug has been reported and will be fixed in the next Emacs release; in any case, building Emacs from source works fine.
When initialized by iBoot, the framebuffer is in x2r10g10b10 mode, which works fine for the boot loaders and Linux but isn’t supported by X.org. Putting the framebuffer into x8r8g8b8 mode works, but it means colors will be off either before or after the switch. In keeping with our “stall when possible” strategy, we use userspace code to switch the framebuffer just before X starts, and accept that the white-on-black text appears as green-on-black before then.
There appears to be a bug somewhere that affects, in particular, videos shown in browser windows, where black sometimes shows up as a solid blue color.
Feel free to contact me at pipcet@gmail.com, on GitHub, or in any of the other usual ways. So far, there has been too little communication rather than too much of it.