jlinder / bincheck

Verify CockroachDB binaries

cockroachdb/bincheck

bincheck verifies the sanity of CockroachDB release binaries. At present, the sanity checks are:

• starting a one-node server and running a simple SQL query, and
• verifying the output of cockroach version.

Testing a new release

Open a PR that updates the version in VERSION on the appropriate branch (eg: release-2.0 for 2.0 builds). You can use the little pencil button to edit the file directly through the GitHub UI (no need to fork the repo); just make sure to select the "start a pull request" option. The PR will automatically kick off checks to verify the release. macOS and Linux binaries are verified by TravisCI; Windows binaries are verified by AppVeyor. If the checks complete successfully, you can merge the PR and you're done!

The nitty-gritty

Overview

For the macOS and Windows binaries, the mechanics involved are simple. We ask Travis or AppVeyor, respectively, to spin us up a macOS or Windows VM, download the appropriate pre-built cockroach binary, and run our sanity checks.

Linux is more complicated, not out of necessity, but out of ambition. We co-opt the Linux verification step to additionally test support for pre-SSE4.2 CPUs^†. This requires emulating such a CPU, and Linux is the only operating system that is feasible to run under emulation. Windows and macOS have prohibitively slow boot times, and, perhaps more importantly, Windows and macOS install media are not freely available.

So, with the help of Buildroot, an embedded Linux build manager, this repository ships an 8.7MB Linux distribution that's capable of running under QEMU and launching our pre-built CockroachDB binaries. To verify the Linux binaries, we first boot this Linux distribution on an emulated pre-SSE4.2 CPU (qemu-system-x86_64 -cpu qemu64,-sse4.2), then run our sanity checks from inside this VM.

^†SSE4.2 support is particularly important in CockroachDB, since RocksDB internally uses a CRC32C checksum to protect against data corruption. SSE4.2 includes hardware support for computing CRC32C checksums, but is only available on CPUs released after November 2008. Producing a binary that can dynamically switch between the SSE4.2 and non-SSE4.2 implementations at runtime has proven difficult.

Updating the Linux image

After installing Buildroot:

$ make qemu_x86_64_glibc_defconfig BR2_EXTERNAL=${BINCHECK-REPO}/buildroot
$ make menuconfig  # Only if configuration changes are necessary.
$ make
$ cp output/images/bzImage ${BINCHECK-REPO}/images/qemu_x86_64_glibc_bzImage

At the time of writing, qemu_x86_64_glibc_defconfig instructed Buildroot to build a Linux 4.9 kernel, install Bash and OpenSSH into userland, and configure sshd to boot at startup and allow passwordless root authentication. /etc/fstab is modified to mount the first hard drive at /bincheck, assuming it's a FAT volume.