b-paul / pac-nix

Nix derivations for UQ PAC's work.

pac-nix

An experiment to package PAC's tools with the Nix package manager.

This repository contains fixed-output derivations which fetch and build a fixed version of each tool and its dependencies. Nix's declarative nature makes this fast and reliable, independent of particular distributions or their package repository quirks. Further, since each dependency is itself a static derivation, these can be cached. z3 and BAP, which previously took dozens of minutes each, are fetched in seconds.

Overall, Nix is an effective tool and well-suited to this task of a consistent meta-build and dependency manager system.

structure

The pkgs.nix file defines the package set as the usual <nixpkgs> extended with those from this repo.

For users of tools, the main packages are:

• aslp: the ASLp partial evaluator with ARM's MRA (provides aslp and aslp-server),
• basil: the Basil tool for analysis and transpilation to Boogie code (provides basil),
• gtirb-semantics: inserts instruction semantics from ASLp into the GTIRB from ddisasm (provides gtirb-semantics, debug-gts.py, and proto-json.py), and
• bap-aslp¹: a version of official BAP with a bundled ASLp plugin (this is the preferred BAP and provides the bap executable), and
• alive2-aslp: a fork of regehr/alive2, using Aslp to provide semantics for translation validation of the LLVM Aarch64 backend (provides backend-tv, others).

These packages are each defined in a .nix file of the same name, then instantiated within overlay.nix and built into a package set in pkgs.nix.

We also package some related third-party tools (without endorsement from their authors):

• ddisasm: GrammaTech's datalog disassembler (provides ddisasm),
• alive2-regehr: a fork of AliveToolkit/alive2, performs translation validation of LLVM's Aarch64 backend by lifting MCInst back to LLVM IR (provides backend-tv, others).

Other packages

These are less frequently used and might be untested.

• bap-primus: PAC's fork of BAP with the Primus Lisp PR but without ASLp (provides bap-primus)
• godbolt: the Godbolt compiler explorer with the Basil toolchain for interactive use (provides godbolt)
• alive2: translation validation of LLVM IR, designed to verify LLVM's middle-end optimisations (currently this version is frozen to llvm-translator's version)

Other Nix files also define dependencies needed by the end-user tools.

usage

first time

First, install the Nix package manager with flakes enabled:

curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install

This should extend your PATH with ~/.nix-profile/bin which is where installed programs will go.

Then, allow your user to use binary caches for faster package installation.

printf '%s\n' "extra-trusted-users = $USER" | sudo tee -a /etc/nix/nix.conf

(Optional) Add an alias for this package repository. This lets you write pac in place of github:katrinafyi/pac-nix in the commands below.

nix registry add pac github:katrinafyi/pac-nix

installing packages

Installing a package is straightforward.

nix profile install github:katrinafyi/pac-nix#aslp  # add -Lv for more progress output
# say 'y' to config settings

For other packages, change the term after # to bap-aslp, basil, etc. This will build and make available an executable on your PATH (stored in ~/.nix-profile/bin).

Note that this will install the package at a particular commit hash from the upstream repository. The next sections will discuss building a package from local sources and setting up development environments.

To list available packages:

nix flake show github:katrinafyi/pac-nix --allow-import-from-derivation --impure

To list installed packages:

nix profile list

To uninstall, use the index from nix profile list in this command:

nix profile remove 1234

To upgrade all packages:

nix profile upgrade '.*'  # or use an index instead for specific packages

Nix is powerful but the documentation is of mixed quality. nix-tutorial introduces some other commands. Otherwise, it will be useful to search as far as you can.

garbage collection

The /nix/store folder can get quite large. nix profile creates a snapshot of your packages (a generation) after each package operation which can keep outdated and uninstalled packages inside the Nix store. You can use these commands to clear shapshots older than 7 days then delete their packages from the store.

nix profile wipe-history --older-than 7d
nix-store --gc

bonus: binary cache

The GitHub Actions workflow maintains a custom binary cache for this repository. Using this cache, you can install these Nix packages while avoiding the need to run any compilations yourself. This can save a fair bit of bandwidth and time.

The cache is served at pac-nix.cachix.org and should be used automatically by the instructions above. This will be visible in its output:

copying path '/nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-ocaml4.14.1-bap-2.5.0' from 'https://cache.nixos.org'...
copying path '/nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-bap-aslp-2.5.0' from 'https://pac-nix.cachix.org'...

local sources / customisation

It is often useful to build a package from a clone of the repository on your local computer (e.g. to test your un-committed changes).

To do this, we can override the src attribute of the corresponding build package. In overlay.nix, some packages have commented overrideAttrs lines. These are the build packages which are wrapped into the user-facing tools.

To use local sources, check out this repository then uncomment the line and change the path to be your local path. (It is important the path is not quoted so Nix handles it correctly.)

When installing packages, you'll need to reference your local clone. For example, from this repository's directory:

nix profile install .#aslp

Further customisation can also be done in the files. The JRE and JDK are currently fixed to Java 17, and you may also add more Nix files or override other attributes.

development environments

With each package, Nix also downloads and stores its dependencies. As such, you may want to re-use these when developing instead of duplicating them into your local system. This may also speed up the getting-started process and make it more reliable.

This is done with the nix develop which starts a subshell within a particular Nix environment.

For example,

nix develop github:katrinafyi/pac-nix#ocaml

spawns a shell with the build environment for our OCaml packages (aslp, gtirb-semantics) and all dependencies installed. From within this shell, you can start IDEs and tools to inherit the environment.

These shells are defined in the *-shell.nix files (e.g. ocaml-shell.nix). You can copy this format to make other shells. inputsFrom defines the package(s) whose dependencies we will load (i.e. what we want to build), and packages are packages to make available for use (e.g. language servers, compilers). After adding packages to packages or inputsFrom, you may need to add them to the derivation argument within the braces { ... }:.

Finally, new shells will need to be given a name in flake.nix's devShells attribute.

See also: nix-shell manual page.

updating packages

Periodically, these Nix files will need to be updated with new changes from upstream. A daily GitHub action will attempt to update all packages. Its most recent status is shown at the top of this README.

The update process is automated by a script:

./update.py do-upgrade  # or `./update.py check` to check only

This will update the hash in each Nix file with the latest then attempt to build the new packages. If successful, this will commit the changes.

The basil derivation is most fragile since it relies on SBT to fetch its dependencies. The depsSha256 will need to be changed manually if the script fails at that point.

miscellany

search.nixos.org is very useful for searching package names.

Nix pills and nix.dev are good resources if you wish to write your own packages. It is also a good idea to browse the nixpkgs monorepo for similar derivations.

Nix can also bundle a package into a standalone executable, AppImage, or Docker image with nix-bundle.

Footnotes

1. Due to the plugin loading method, bap-mc -- [bytecode] will not work to disassemble one opcode. Instead, you should omit the -- or pipe the bytes via stdin echo [bytecode] | bap-mc. ↩