Giters

jgarte / spirit-of-nix

Presentation on some of the technical values of the Nix ecosystem

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Spirit of Nix

Who are you and why do you use Nix?

I assume you may use:

  1. Nix - maybe as a pacman or apt replacement - I.E. nix-env -iA lldb
  2. Flakes some - maybe via a dev tooling setup like github:srid/haskell-flake or github:ipetkov/crane
  3. NixOS via nixos-rebuild and /etc/nixos/configuration.nix or /etc/nixos/flake.nix

You probably like Nix for the:

  1. declarative specification of package sets and systems
  2. reproducible dev and operating system environments
  3. flexibility to patch/override software without the intervention of a sysadmin or IT
    • (without losing track of what was installed where and how!)

And I’d like to give you a leg up so you can

  1. Give you a feel for the values of the Nix community
  2. Understand some idioms and some things I wish I had known when I started using Nix more seriously

Shoutouts to Gabriella Gonzalez who already literally wrote the book

If this talk at all interests you go read NixOS in Production - it will get you straight from zero NixOS running your services complete with slick tooling and polished user experience.

Values of the Nix/NixOS community and technology

This is worth bearing in mind as you assess the usefulness of these tools in production.

It can be very painful to adopt a technology without first aligning your values with that of the community/technology. It can end very painfully if you do not - see Node.js and Joyent for a “fun” example.

Tech value 1: “The Zen of Nix”

Every common build/test/deploy-related activity should be possible with at most one command using Nix’s command-line interface – Gabriella Gonzalez, Nixos in Production

This is a value that seems lovely and useful on the surface - and is in fact probably the biggest selling point of Nix as a productivity tool.

I certainly love this tenet.

However you may chafe with this value if you tend to reach for scripts, automation, or some favorite tool as your first solution.

What this value means in practice is that you will be writing more Nix code than shell, haskell, elixir, rust, lua, go (insert your favorite language here) to coordinate package, process, and system definitions.

This zen atomicity of nix workflows, in fact, stems from Nix’ properties as a pure, lazy, functional programming language.

This is a silly thing to say but: if you do not like Nix the programming language, you will likely not enjoy the experience of using the Nix ecosystem in anger.

Tech value 2: The freedom and responsibility to patch

Nix makes it very easy to locally patch, override, or otherwise modify software you use. In some regards it embodies the tenets of Free software with capital FSF.

Combined with the other benefits of Nix (i.e. ephemeral environments, atomic upgrades, multiple non-interfering package installs) - you have the power to really fearlessly hack your own environment.

This “power to” also often translates to an “expecation that” you roll up your own sleeves.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND. – nixpkgs, COPYING

The probability that you will submit a patch to nixpkgs in the process of deploying your first production system is very close to 1.

Tech value 3: The Golden Rule of Software Distributions

A corollary to the freedom and responsibility to patch

Thanks again to Gabriella for putting words to this ethic.

A locally coherent package manager requires a globally coherent software distribution. – Gabriella Gonzalez, “The Golden Rule of Software Distributions”

Even though Nix provides you with great power to patch software locally, nixpkgs as a coherent package set is perhaps the Nix community’s greatest contribution to the wider software community.

If you were around during the days of “Cabal hell” or have ever experience PyPi or NPM hell, then you know the value of a package ecosystem that is tested altogether.

This relates to the freedom and responsibility to patch in the following way:

You will occasionally feel the desire to work around bugs in nixpkgs (or often upstream in the software itself). This is easy to do in the short run, but you will quickly rue the decision to vendor too many definitions, override too many packages, or locally patch too many things.

Corollary: Maximum sharing

The technical advantage of upstreaming things and a globally coherent package set.

The way Nix gets any good performance at all is by reusing the same definitions across the entire package set.

For instance: dynamic linking is strongly preferred in Nix and vendoring is frowned upon. This is because static linking vastly increases the size of the nix store.

Nixpkgs and NixOS idioms

When working with the Nixpkgs package set you will run into these types of nix expressions often. Understanding them will help you get the most out of your experience as a contributor and as a user.

Note I’m going to assume you know what a derivation is, but just to recap: a derivation is the build recipe that go in the nix store as .drv extension files. Nix the language is “pure” with respect to derivations. When you as to build a derivation, it is interpreted into the nix building process and the result is “realized” to the final outputs (i.e. take the recipe for bash and turn it into the bash program).

Fixed points

“modification in place” is required to give you the freedom to patch and modify your software. But as a pure functional language, no such feature exists. The solution to this is a variety of fixpoints.

The two primary ones are:

  • overlays - used to configure the nixpkgs package set
  • eval-modules (aka eval-config, aka nixos) - used to configure NixOS

Note that when using nixos or writing an overlay, you are programming to an interface - morally writing the (a -> a) function where fix :: (a -> a) -> a.

Overlays

I won’t go over overlays too much because the wiki and nixos.org docs on them are good and you probably used them some already. However a couple subtleties might be noted.

  • Referring to final at the top-level is a good way to get infinite recursion
# first parameter is the final result of the fixpoint, prev is the
# result just prior
final: prev: {
  redis = final.redis.overrideAttrs ... # <- kaboom!
}
  • Composing overlays can be error prone due to non-commutativity of overlays!
{
  description = ''
    how to overwrite a thing you wanted on accident,
    beware of this!
  '';
  outputs = { nixpkgs, ... }: {
    # composeManyExtensions can be good but beware of ordering
    # problems!
    overlays.default = nixpkgs.lib.composeManyExtensions [
      (final: prev: { redis = /* do some very important redis modifications here ... */; })
      (final: prev: { redis = /* overwrite your work here! */;  })
    ];
  };
}

eval-modules (aka nixos)

The eval-modules fixpoint has a type you’ve used before if you have configured NixOS:

config is the final result of the configuration, a-la final of overlays, however it is much more common to refer to it in configs:

{ config, pkgs, ... }: {
  # Again, this is slideware, don't set TERMINFO_DIRS unless you know
  # what you are doing, it is set upstream pretty well
  config.environment.variables.TERMINFO_DIRS =
    [ "${pkgs.ncurses}/share/terminfo" ];
}

This means that infinite recursion is much more likely to occur in NixOS configs. Sadly, this can be miserable to debug, but the situation is improving.

Writing modules

Seeing as how you want to put Nix into production, you are undoubtedly going to write your own NixOS modules. For this I can offer no better place than the nixos.org docs.

Aside on deployment tools

I humbly posit that NixOS is the killer feature of Nix (aside from dev environments, but that’s child’s play). There are a nice variety of tools (like terraform-nixos-ng) to deploy NixOS systems to various cloud providers or a server of your ownership. I recommend nixos-rebuild as a starting place, it’s pretty good as a deployment tool.

callPackage

The following comes from pkgs/build-support/ocaml/dune.nix.

This expression is meant to be used like so:

callPackage pkgs/build-support/ocaml/dune.nix { }

It defines the buildDunePackage function that most all dune-based ocaml packages are built with.

{ lib, stdenv, ocaml, findlib, dune_1, dune_2, dune_3 }:

{ pname, version, nativeBuildInputs ? [], enableParallelBuilding ? true, ... }@args:

let Dune =
  let dune-version = args.duneVersion or "3"; in
  { "1" = dune_1; "2" = dune_2; "3" = dune_3; }."${dune-version}"
; in

if (args ? minimumOCamlVersion && lib.versionOlder ocaml.version args.minimumOCamlVersion) ||
   (args ? minimalOCamlVersion && lib.versionOlder ocaml.version args.minimalOCamlVersion)
then throw "${pname}-${version} is not available for OCaml ${ocaml.version}"
else

stdenv.mkDerivation ({

  inherit enableParallelBuilding;
  dontAddStaticConfigureFlags = true;
  configurePlatforms = [];

  buildPhase = ''
    runHook preBuild
    dune build -p ${pname} ''${enableParallelBuilding:+-j $NIX_BUILD_CORES}
    runHook postBuild
  '';
  checkPhase = ''
    runHook preCheck
    dune runtest -p ${pname} ''${enableParallelBuilding:+-j $NIX_BUILD_CORES}
    runHook postCheck
  '';
  installPhase = ''
    runHook preInstall
    dune install --prefix $out --libdir $OCAMLFIND_DESTDIR ${pname} \
     ${if lib.versionAtLeast Dune.version "2.9"
       then "--docdir $out/share/doc --mandir $out/share/man"
       else ""}
    runHook postInstall
  '';

  strictDeps = true;

} // (builtins.removeAttrs args [ "minimalOCamlVersion" "duneVersion" ]) // {

  name = "ocaml${ocaml.version}-${pname}-${version}";

  nativeBuildInputs = [ ocaml Dune findlib ] ++ nativeBuildInputs;

  meta = (args.meta or {}) // { platforms = args.meta.platforms or ocaml.meta.platforms; };

})

Overriding a callPackage function

callPackage is a function that “auto-applies” a function (or filepath that evaluates to a function) to the parameters if they exist in the current package environment. This is commonly at the top-level package set like this expression, but may also be applied to other environments so python or haskell derivations also can have a similar type.

Without any overrides, this would use the top-level stdenv, lib, ocaml, etc from the top-level package set. But, say you wanted to overide a particular dependency (findlib for instance), it could be “overridden” like this:

overridenBuildDunePackage =
  prev.buildDunePackage.override { findlib = my-other-findlib; };
overridenBuildDunePackage =
  callPackage ./pkgs/build-support/ocaml/dune.nix { findlib = my-other-findlib; };

This overrides the inputs to the callPackage function - which is often where “official” extension points are defined:

The following comes from pkgs/tools/compression/bzip2/default.nix and it has official overrides for enableStatic or enableShared

{ lib, stdenv, fetchurl
, enableStatic ? with stdenv.hostPlatform; isStatic || isCygwin
, enableShared ? true
, autoreconfHook
, testers
}:
  stdenv.mkDerivation { ... }

overrideAttrs

The overrideAttrs function is the idiomatic way of overriding the fields of a derivation. It is a field of all derivations built with stdenv.mkDerivation (in other words, most all derivations) - a “method” if you will.

You will often use this as part of an overlay to modify the build process, configure flags, patches, etc of an existing definition.

final: prev: {
  # overrideAttrs : Derivation -> (Derivation -> Attrs) -> Derivation
  #                 ^ self        ^ overriding function
  redis = prev.redis.overrideAttrs (o: {
    patches = (o.patches or [ ]) ++ [
      # fetchpatch is an invaluable resource, just beware of using it
      # on "bootstrap" packages which can cause infinite loops
      (prev.fetchpatch {
        url = "https://github.com/example/example/pulls/1.patch";
        # When first building a fixed-output-derivation (one with a
        # content hash) you can run into cache-freshness issues. So on
        # the first build use the bogus hashes (like this one) to get
        # the correct hash from the error message
        hash = "sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=";
      })
    ];
  });
}

Note that often you will want to make a wrapper around an existing program to avoid rebuilding the whole thing, that’s when makeWrapper and runCommand come in handy:

{ config, pkgs, ... }: {
  config = {
    # Slideware! defaultPackages is a bit error-prone! I recommend
    # nix-shells/nix develop
    environment.defaultPackages.my-psql =
      let
        pgUrl =
          "postgres://${config.services.my-pg.user}@${config.services.my-pg.host}:${config.services.my-pg.port}/${config.services.my-pg.database}";
      in
        pkgs.runCommand "my-psql-with-url"
          { nativeBuildInputs = [ pkgs.makeWrapper ]; } ''
            mkdir -p $out/bin
            makeWrapper ${prev.postgresql}/bin/psql $out/bin/psql \
              --set-default POSTGRES_URL ${pgUrl}
          '';
  };
}

Schism. Err, I mean flakes!

Why are flakes so controversial?

Simple, they contradict some of the aforementioned values, namely the Golden Rule of Software Distributions. Flakes allow for much more local overriding and encourage a more local-overriding approach.

There are other issues, too: some flake-specific features still haven’t ported back to standard Nix (i.e. evaluation caching). Plus, the development of flakes broke from the development of standard Nix evaluation in ways that feel like standard Nix use was forgotten.

On top of that, using flakes requires that you already know the idioms mentioned above, and more! That’s why I would focus on being good at Nix the language and ecosystem - since flakes build on them.

That said, there are good things about flakes!

Though the UX of them is a bit obtuse. Here are a few things I wish I’d known:

  • flake refs and “new style” commands

    Like nix-shell shell.nix but for the environment of the derivation:

    #              attributes of flake outputs follow the '#' (what these
#              are are shifting and probably will require
#              experimentation to get the thing you want,
#              unfortunately)
#              v
nix develop ./#packages.python3Packages.aiohttp
#           ^ flake in current directory, but could be git+https or
#           other url

    nixos-rebuild can use flakes:

    nixos-rebuild switch --flake git+https://git.sr.ht/~jsoo/dotfiles#nixosConfigurations.vbox
  • flake urls query params are nice! 
    {
  inputs.jsoo.url = "git+https://git.sr.ht/~jsoo/dotfiles?rev=13aadf70cba83b631604e947c5aad5fb3f639f0c";
}
  • The wiki on flakes is still the best reference on the current flake schema (which changes, frustratingly, as mentioned)

Resources

About

Presentation on some of the technical values of the Nix ecosystem

License:Creative Commons Attribution Share Alike 4.0 International

Languages

Language:Nix 100.0%