mtrsk / nixos-ami-building

Methods for building custom NixOS AMIs

Methods for building custom NixOS AMIs

This project demonstrates various ways to build a custom NixOS AMI from a configuration.nix file.

It largely exercises the methods shown in jackkelly's blog post http://jackkelly.name/blog/archives/2020/08/30/building_and_importing_nixos_amis_on_ec2/.

Overview

This README presents the following methods to create AMIs:

• Methods to build an AMI image file on your computer and then upload it:
  • Method 1: nix-build
  • Method 2: nixios-generators These methods build your image from scratch.
• Methods to build an AMI image on an EC2 machine:
  • Method 3: packer This method starts with an official NixOS AMI, applies your configuration.nix, and snapshots it into a new AMI.

Prerequisites

All methods assume that you have:

• checked out the version of nixpkgs you want to use at $HOME/src/nixpkgs.
• Adjusted ./nixos/configuration.nix to contain your desired NixOS config.

Method 1: nix-build

NIX_PATH=nixpkgs=$HOME/src/nixpkgs \
  nix-build --no-out-link $HOME/src/nixpkgs/nixos/release.nix \
  --arg configuration nixos/configuration.nix \
  -A amazonImage.x86_64-linux

This prints to stdout the path to a dir containing the AMI .vhd file.

Method 2: nixos-generators

nixos-generators is a small repo of wrapper scripts to build NixOS images with simple commands.

NIX_PATH=nixpkgs=$HOME/src/nixpkgs \
  $(nix-build --no-out-link https://github.com/nix-community/nixos-generators/archive/master.tar.gz)/bin/nixos-generate \
  -f amazon

This prints to stdout the path to a subdir of the dir containing the AMI (you have to strip the last 2 dirs off it to get to the .vhd file's dir).

The inner nix-build builds nixos-generators itself, and then we invoke its nixos-generate binary. It picks up our nixos/configuration.nix by file naming conventions.

Method 3: packer

In contrast to the previous methods, this requires having set up ~/.aws/credentials because it will do the AMI building on a throwaway EC2 machine that packer launches. It starts the official NixOS AMI, applies your configuration.nix, and snapshots the result into a new AMI. You can read a basic Packer tutorial here.

I've saved jackkelly's Packer config in ./packer/nixos-packer-example.json, and modified it to give the machine that does the AMI building a larger disk, to not run out of disk space (making sure that the device_name is the one for the root device of that instance type). The config uses "most_recent": true to select the most recent official NixOS image as a base.

In the below, replace AWS_REGION=eu-central-1 by the region to create the AMI in.

NIX_PATH=nixpkgs=$HOME/src/nixpkgs \
  nix-shell --pure -p packer --run 'cd packer/ && AWS_REGION=eu-central-1 packer build nixos-packer-example.json'

With the Packer method you do not need to import the AMI into EC2; Packer already does that for you.

Importing built AMIs into EC2

You need an S3 bucket to upload the AMI to. jackkelly's blog post kindly provides a CloudFormation template; I saved it saved in this repo under ./ami-importing/cloudformation/template.yaml.

I then downloaded this version of the nixpkgs create-amis.sh script of create-amis.sh and modified it

to comment out the calls to make_image_public, and also comment out the loop in upload_all that iterates across the regions

The result is in this repo under ./ami-importing/create-amis.sh.

Steps to upload the AMI:

1. Open CloudFormation in the AWS console, and import ./ami-importing/cloudformation/template.yaml into it. This creates an S3 bucket for you.

• Be aware that the bucket is created in the AWS region that you've currently chosen (in the top right). Note down that region.

1. Open S3 in the AWS console, and figure out the name of the created S3 bucket. It should look like nixos-ami-building-vmimportbucket-prf5j5yydsx3.
2. Edit ./ami-importing/create-amis.sh, replacing:

• the home_region= variable by the region that you created your bucket in
• the bucket= variable by your bucket name

1. Configure your ~/.aws/credentials file so that the aws CLI utility has access to your AWS account. (This is outside the scope of this README, but easy to find docs for.)

2. Run the script (replace the path to the dir containing the AMI .vhd by your own):

   NIX_PATH=nixpkgs=$HOME/src/nixpkgs \
     ami-importing/create-amis.sh \
     /nix/store/1wgfx4m76bcmdyzsw49pvs13l6gl6gim-nixos-amazon-image-20.09beta-111781.gfedcba-x86_64-linux/

   Click to expand issues of the upload script itself

   Note that as of writing, like most shell scripts, the script doesn't do proper error handling: On success, no exit code is set, and it continues to run subsequent steps even if earlier steps failed because you haven't configured AWS credentials. Don't do shell scripts.

After this, you should find the AMI in your EC2 AMI list.

Note that the create-amis.sh remembers in the configured state_dir= which images were already uploaded. If you want to re-build an AMI from the same nixpkgs version (but e.g. different configuration.nix), you have to delete ("Deregister") the AMI on EC2, and wipe state_dir= (thus, by default, rm -r ~/amis). Otherwise the script will be a no-op.

Troubleshooting

Click to expand Troubleshooting section

• The given S3 object is not local to the region
  • The home_region= in the upload script does not match your bucket's region.
• Upload error when calling the CreateMultipartUpload operation: Access Denied
  • You either have not configured ~/.aws/credentials, or bucket= in the upload script does not match your bucket's name.
• What does a successful create-amis.sh invocation look like? Roughly like this:

  Image Details:
   Name: NixOS-20.09.git.6608ea8eb6a-x86_64-linux
   Description: NixOS 20.09.git.6608ea8eb6a x86_64-linux
   Size (gigabytes): 3
   System: x86_64-linux
   Amazon Arch: x86_64
  Checking for image on S3
  Image missing from aws, uploading
  upload: ../../../../nix/store/6blb4hdpjyfi0ysd169sw7y7p17l3idh-nixos-amazon-image-20.09beta-111781.gfedcba-x86_64-linux/nixos-amazon-image-20.09beta-111781.gfedcba-x86_64-linux.vhd to s3://nixos-ami-building-vmimportbucket-prf5j5yydsx3/nix/store/6blb4hdpjyfi0ysd169sw7y7p17l3idh-nixos-amazon-image-20.09beta-111781.gfedcba-x86_64-linux/nixos-amazon-image-20.09beta-111781.gfedcba-x86_64-linux.vhd
  Importing image from S3 path s3://nixos-ami-building-vmimportbucket-prf5j5yydsx3/nix/store/6blb4hdpjyfi0ysd169sw7y7p17l3idh-nixos-amazon-image-20.09beta-111781.gfedcba-x86_64-linux/nixos-amazon-image-20.09beta-111781.gfedcba-x86_64-linux.vhd
  Waiting for import task import-snap-0054ec2e7d6bf866d to be completed
   ... state=active progress=2 snapshot_id=null
   [..]
   ... state=active progress=94 snapshot_id=snap-0054ec2e7d6bf866d
   [..]
   ... state=completed progress=null snapshot_id=snap-0054ec2e7d6bf866d
  Registering snapshot snap-0054ec2e7d6bf866d as AMI
  {
    "eu-central-1.x86_64-linux": "ami-06743a1c5bc56e348"
  }
  

Attribution

This work was sponsored by Tyler Technologies and FP Complete.