pypi2nix
is a command line tool that generates Nix expressions from
different requirements.txt
. This is useful for:
Building a Nix derivation for a program written in Python as part of packaging it.
Setting up a development environment to hack on a program written in Python.
The only way we can fix bugs with pypi2nix is if you report them. Please create an issue if you discover problems.
pypi2nix
will (until further notice) only work with latest unstable
channel. This is due to ongoing changes in Python infrastructure happening in
Nixpkgs.
The Nixpkgs manual section about Python makes good reading if you haven't seen it already.
Contents
pypi2nix
is part of nixpkgs
. If you just want to use
pypi2nix
on your system, it is recommended that you install it via
the regular means, e.g. nix-env -iA nixos.pypi2nix
on NixOS or
nix-env -iA nixpkgs.pypi2nix
on other systems utilizing nix.
Make sure Nix is installed:
% curl https://nixos.org/nix/install | sh
Currently pypi2nix
is only tested against linux
systems.
Supported nixpkgs
channels are nixos-19.09
and
nixos-unstable
. Due to the nature of nixos-unstable
the
occasional breakage of pypi2nix
is to be expected. We try to
provide fixes in that regard in a timely manner.
For just installing the package with a command, use nix-env:
git clone https://github.com/nix-community/pypi2nix.git cd pypi2nix nix-env -if .
If you prefer to explicitly declare each installed package in your Nix(OS) or project configuration, you can do the following:
First, import the package from its default.nix
by fetching the
whole git repository with pkgs.fetchgit
. Afterwards you can just
add the imported attribute the list of installed software.
Below you find an example for NixOS' configuration.nix
. Other
methods like home-manager work similar:
let pypi2nix = import (pkgs.fetchgit { url = "https://github.com/nix-community/pypi2nix"; # adjust rev and sha256 to desired version rev = "v2.0.1"; sha256 = "sha256:0mxh3x8bck3axdfi9vh9mz1m3zvmzqkcgy6gxp8f9hhs6qg5146y"; }) {}; in environment.systemPackages = [ # your other packages pypi2nix ];
The easiest way to generate a Nix expressions is to invoke:
% pypi2nix -e packageA -e packageB==0.1
If you also have a requirements.txt
file for your Python project you can use
the -r
option.
% pypi2nix -e packageA -e packageB==0.1 \ -r requirements.txt -r requirements-dev.txt
Option -V
tells pypi2nix which python version to be used. To see which
Python versions are available consult pypi2nix --help
.
Once Nix expressions are generated you should be able to see 3 new files:
requirements_frozen.txt
- full frozen set for your for your pypi2nix call. This is the output you would expect frompip freeze
.requirements.nix
is a file which contains a nix expression to for the package set that was built.requirements_override.nix
- this is an empty file which lets you override generated nix expressions.
Build one package:
% nix build -f requirements.nix packages.empy
Build all packages:
% nix build -f requirements.nix packages
Build python interpreter with all packages loaded:
% nix build -f requirements.nix interpreter % ./result/bin/python -c "import empy"
Enter development environment:
% nix run -f requirements.nix interpreter [user@localhost:~/dev/nixos/pypi2nix) % python -c "import empy"
If you are working on a project where its dependencies are defined in
requirements.txt
then you can create a default.nix
and add generated
packages as buildInputs
, as demonstrated here:
{}: let python = import ./requirements.nix { inherit pkgs; }; in python.mkDerivation { name = "ProjectA-1.0.0"; src = ./.; buildInputs = [ python.packages."coverage" python.packages."flake8" python.packages."mock" python.packages."pytest" python.packages."pytest-asyncio" python.packages."pytest-cov" python.packages."pytest-mock" python.packages."pytest-xdist" python.packages."virtualenv" ]; propagatedBuildInputs = [ python.packages."aiohttp" python.packages."arrow" python.packages."defusedxml" python.packages."frozendict" python.packages."jsonschema" python.packages."taskcluster" python.packages."virtualenv" ]; ... }
As you can see you can access all packages via python.packages."<name>"
. If
you want to depend on all packages you can even do:
propagatedBuildInputs = builtins.attrValues python.packages;
-v
- Increase amount and detail of information output to the user.
Verbosity levels are
ERROR
,WARNING
,INFO
andDEBUG
in that order. The default verbosity isINFO
. -q
- Reduce amount and detail of information output to the user. See
-v
for more information. -I/--nix-path TEXT
- Add entries to the
NIX_PATH
environment variable similarly to how-I
works withnix
executables likenix-build
. This can be useful for generating package sets based on a differentnixpkgs
version than the one used one the local system. --nix-shell PATH
- Path to an alternative version of the
nix-shell
command. The default is the first executable that will be found in the currentPATH
of the system. --version
- Show the current version of
pypi2nix
--basename TEXT
- This option determins the name the produced files. So with
--basename environment
you would get the filesenvironment.nix
,environment_frozen.nix
andenvironment_override.nix
. --extra-build-inputs/-E TEXT
- Extra build inputs that the required python packages need to run,
e.g.
libffi
orlibgl
. In that case you would provide-E "libffi libgl"
. These nix packages will be available in the build environment for the wheels. --emit-extra-build-inputs/--no-emit-extra-build-inputs
- These options let you control if external build dependencies
specified via
-E
will end up in the generated nix package set. Please note that if you select this option, your overrides need to make sure that python packages find their respective external dependencies. --extra-env/-N TEXT
- Extra environment variables that will be passed to the build
environment. Note that you can use nix expressions in this
string, e.g.
-N 'BERKELEYDB_DIR=${pkgs.db.dev}'
. --enable-tests/-T
- Specify this flag if you want to enable the check phase of all packages in the generated nix expression. Please note that this feature is highly exprimental and will probably not work for your use case.
--python-version/-V
- Specify the python version you want the requirement set to be
built with. The default is
3
which translates to thepython3
derivation ofnixpkgs
. --requirements/-r FILE
- Specify a requirements file, similar as you would with
pip
.pypi2nix
tries to be fully compatible with the file format ofpip
. --editable/-e TEXT
- This option allows you to specify individual requirements that get
added to the requirement set, e.g.
pypi2nix -e attrs
,pypi2nix -e $HOME/src/myproject#egg=myproject
orpypi2nix -e .#egg=myegg
. --setup-requires/-s TEXT
- Allows you to specify python packages that need to be present in
the build environment of other packages, a good example of this
would be
setuptools-scm
. Note thatpypi2nix
tries to detect these dependencies on its own. You only need to specify this flag in cases where a package author or maintainer forgot to mention build time dependencies in their setup or neithersetup.cfg
norpyproject.toml
is used. --overrides/-O URL
- Allows you to specify additional overrides that conform to the
general structure of
requirements_override.nix
. We support regular URLs withhttp
andhttps
scheme and alsogit
. An example for usinghttps
would bepypi2nix -O https://myoverrides.test/overrides.nix
. Reusing an overlay from a git repository would be done like so:pypi2nix -O git+https://github.com/nix-community/pypi2nix.git&path=requirement_override.nix
. Please keep in mind that these overrides are incorporated in a nix expression with a precalculated hash value. So if the file changes upstream your generated package can not be built anymore. --default-overrides/--no-default-overrides
- Pull in overrides from
https://github.com/nix-community/pypi2nix-overrides
. This feature is enabled by default. --wheels-cache/-W TEXT
- A location where prebuilt wheels can be found. This option will
ultimately be passed to
pip --find-links
. Only point to wheels that are built throughpypi2nix
on your own or a very similar system. --build-directory TEXT
Warning A bug in
pypi2nix
currently prevents some packages from being built with this option set. It is recommended to not use this flag.The directory where pypi2nix would build the python environment to generate the desired nix expression. If not specified, the build directory will be temporary and is deleted before the program exits.
I hope nobody is expecting pypi2nix
to do always a perfect job. In Python
packaging, there are just too many different cases that we will never be able to
cover. What pypi2nix
tries to do is to get you very close.
Sometimes pypi2nix
fails entirely. If this happens, open a bug --
it's almost always a bug in pypi2nix
. However, sometimes
pypi2nix
succeeds but the resulting requirements.nix
file
fails during the building of your Python package. Depending on what
the problem is, this section may be helpful.
Quite a few Python packages require non-Python dependencies to be
present at build time. These packages will fail to build with error
messages about not being able to find foo.h
or some fooconfig
file. To work around this, pypi2nix
has -E
options which can
be used to include extra non-Python dependencies.
For example, psycopg2
requires pg_config
binary to be present at installation time:
% pypi2nix -v -V 2.7 -e psycopg2 -E postgresql
lxml
requires libxml2
and libxslt
system package:
% pypi2nix -v -V 2.7 -e lxml -E libxml2 -E libxslt
Some packages expect additional environment variables to be set:
% pypi2nix -v -V 2.7 -e bsddb3 -N 'BERKELEYDB_DIR=${pkgs.db.dev}'
Some other failures might be caused because the derivation that
pypi2nix
wrote was incomplete. A very common situation is that
pypi2nix
didn't include all the dependencies of some package. As
an example, execnet
depends on setuptools-scm
, but
pypi2nix
may not detect this.
When this happens, Nix will fail to build execnet
, perhaps with an
error message from distutils/setuptools complaining that it can't find
a distribution for setuptools-scm
. What's happening here is that
normally execnet
would fetch setuptools-scm
from PyPI, but Nix
disables network access to guarantee reproducability. So when you
build execnet
, it fails to find setuptools-scm
.
For these situations, pypi2nix
provides a
requirements_override.nix
file, which lets you override anything
that it generated. You can even add new packages to the dependency set
this way.
As an example, let's add setuptools-scm
as a build-time dependency
of execnet
. Here's the requirements_override.nix
:
{ pkgs, python }: self: super: { "execnet" = python.overrideDerivation super."execnet" (old: { buildInputs = old.buildInputs ++ [ self."setuptools-scm" ]; }); }
In a similar way, you can add or remove any Python package.
In addition to the empty autogenerated requirements_overrides.nix
file, you can include pre-existing overrides files. These overrides
will be included the same way as your requirements_overrides.nix
.
The pypi2nix
author also maintains a set of "default" overrides at
https://github.com/nix-community/pypi2nix-overrides/blob/master/overrides.nix --
you can include these by using the --default-overrides
argument to
pypi2nix
. These overrides are designed in such a way that they
only override dependencies that were already present in your
requirements.nix
.
You can also include an overrides file using the -O
command line
argument. pypi2nix
can fetch these overrides from a local file or
over certain common protocols.
http
andhttps
pypi2nix -V 3 --overrides https://raw.githubusercontent.com/nix-community/pypi2nix-overrides/master/overrides.nix
Note that the generated Nix expression will check if contents of the overrides file differs from when the Nix expression was built, and fail if this was the case (or the file does not exist anymore).
- Local files
pypi2nix -V 3 --override ../some/relative/path --override /some/absolute/path
- Git repositories
pypi2nix -V 3 --override git+https://github.com/nix-community/pypi2nix.git#path=overrides.nix
If you want to import a file from a specific git repository you have to prefix its URL with
git+
, quite similar to how you would do in arequirements.txt
file forpip
.
Nothing speaks better than an example:
{ }: let pkgs = import <nixpkgs> {}; python = import ./requirements.nix { inherit pkgs; }; in python.mkDerivation { name = "projectA-1.0.0"; src = ./.; buildInputs = [ python.packages."coverage" python.packages."flake8" python.packages."mock" python.packages."pytest" python.packages."pytest-asyncio" python.packages."pytest-cov" python.packages."pytest-mock" python.packages."pytest-xdist" ]; propagatedBuildInputs = [ python.packages."aiohttp" python.packages."arrow" python.packages."defusedxml" python.packages."frozendict" python.packages."jsonschema" ]; checkPhase = '' export NO_TESTS_OVER_WIRE=1 export PYTHONDONTWRITEBYTECODE=1 flake8 src/ py.test --cov=src -cov-report term-missing coverage html ''; }
Important to know here is that you instantiate all generated packages
as python = import ./requirements.nix { inherit pkgs; };
which
gives you a Python environment with all the packages generated by
pypi2nix
as well as some common utilities.
To create a package you use python.mkDerivation
which works like
the pythonPackages.buildPythonPackage
function in nixpkgs
. All
generated packages are available as one attribute set under
python.packages
.
One of future goals of pypi2nix
project is to also improve the UX of our
Python tooling in nixpkgs. While this is very hard to do within nixpkgs
it
is almost trivial to experiment with this outside nixpkgs
.
A working example is worth 1000 words.
overlay.nix:
self: super: { customPython = (import ./requirements.nix { pkgs = self; }); }
shell.nix:
with (import <nixpkgs> { overlays = [ (import ./overlay.nix) ]; }); customPython.interpreter
Clone pypi2nix repository and using nix-shell
command enter development
environment.:
% git clone https://github.com/nix-community/pypi2nix % cd pypi2nix % nix-shell
Code is located in src/pypi2nix
.
Pypi2nix comes with two kinds of tests: unit tests and integration
tests. They can be found in the folders /unittests
and
/integrationtests
respectively.
Unit tests are straight forward. They are run via pytest and (try
to) follow pytest best practices. Idealy all of pypi2nix's code
should be covered by unittests. If possible unittests should not go
online and fetch data from the internet. If this cannot be avoided
use the @nix
decorator, found in unittests.switches
to mark
tests that require network access.
Integration tests are a little bit more involved. We implemented a
small framework to write new tests and maintain old ones. Check out
integrationtests.framework
for information on how to write custom
integration tests. To run all integration tests run
run_integration_tests.py
from the scripts
directory. If you
use nix-shell
to create your development environment then the
scripts
directory should be in you PATH
variable.
Please note that all integration test cases are classes deriving from
integrationtests.framework.IntegrationTest
. Also all these tests
must end with TestCase
, e.g. MyCustomTestCase
.
The scripts
folder contains programs that help to maintain the
repository. We expect the user to have all the packages from the
build environment of pypi2nix installed. We register the scripts
directory in the users PATH
if they choose to enter nix-shell
in
the top level directory of this project.