nixcrates

A drop-in replacement for cargo which yet uses crates.io index.

Warning: crates.io-index uses 'semantic versioning' (see http://semver.org/) and we don't support this here. 'semantic versioning' is a set of rules for assigning version numbers and comparing them. Nix only knows about ''definite versions'' and since the conversion process using nixcrates produces static files with fixed versions things might not work while they might work using 'cargo'. Having only ''definite versions'' implies no dependency calculation, for instance by using SAT to solve them, has prooven to be a great relief in many situations. However, it would require a cargo2nix tool to have that.

Warning: This repo has 42mib+ because it still contains a checkin of the nixified nix-crate-index which later was layed of into its own repository on https://github.com/nixcloud/nix-crates-index.git. Someone has to do a 'git rebase' and remove it...

Usage

We describe an initial procedure to keep old revisions around.

Iterative update

If you are using our index from https://github.com/nixcloud/nix-crates-index.git then you need to do this:

update the crates.io-index repository:

 cd crates.io-index
 git pull
 git rev-parse master
 fc336466a7ede6f14ce427deb5a64c5d02d14be0

use nixcrates to update it

 cd nix-crates-index/
 # remove all files (nixcrates would not do that for us!)
 rm -Rf *
 nixcrates ../crates.io-index/ ./
 git add *

 # just use the rev from the crates.io-index below        
 git commit -m 'fc3364: new revision added'

afterwards try to build your packages and eventually update nix-crates-index/all-carg-packages.nix to reflect dependency changes
```
 nix-build -A allTargets
```

Initialization

Creating a initial crates.io-index conversion:

nix-shell -p cargoc crates
git clone https://github.com/rust-lang/crates.io-index
git clone https://github.com/nixcloud/nixcrates.git
cd nixcrates
git clone https://github.com/nixcloud/nix-crates-index.git
cargo run ../crates.io-index/ ./nix-crates-index/

This will then create a 1:1 directory structure and nixify all crates.io JSON files.

Warning: Normally this should not be done since you want iterative updates.

Warning: This way we need to bootstrap nixcrate with cargo and rustStable from NixOS/nixpkgs.

Build examples

Currently we use rustc nightly from most recent nipxkgs master!

Example targets:

git clone https://github.com/nixcloud/nixcrates.git
cd nixcrates
nix-build default.nix -A nixcrates
nix-build default.nix -A allTargets
nix-build default.nix -A allCrates.rustache

Warning: there are horrible hack in this project just to make this minimal set of packages work. we were under a lot of time-pressure and low in funding. on judgement day, hopefully god didn't see it!

How it works

We are parsing the json files of the crates.io repository and translating them into nix files. Considering dependencies we are currently only supporting versioning via ~ and ^ as well as single versions. If anything else is used we are switching to the newest version. In case of ^ and ~ we are using the highest allowed version.

How do versions look like in the created

{package_name} is pointing on the newest version. all__{package_name} contains a set of all versions.

There are also several shortcuts to address the newest subversion. For every missing semver version number simply the newest is used. For example all__{package_name}.{package_name}_0_1 points the the version of {package_name} with 0.1.X where X is the highest number found.

What failes and why?

Missing dependencies

For some reason sometimes the crates.io json file is not listing all the dependencies. If this occurs the missing dependency has to be added by hand. However editing an autogenerated file is a pain. Therefore this can be done in the all_carg-packages.nix file. At the bottom of the file you can append additional dependencie. For this see the rusqlite crate example where we added pkg-config.

Resolving dependencies

Currently we are resolving semver only by choosing the highest allowed version out of the cartes.io repository. This works for our targets called allTargets. However here is a constructed case that would fail:

For example your project has 2 dependencies {dep_a} and {dep_b}. Lets assume:

{dep_a} depends on {dep_c} with 0.1.1 <= version <= 0.2.0
{dep_b} depends on {dep_c} with 0.0.1 <= version <= 0.1.2

Now our repo would compile dep_a with dep_c_0_2_0 and dep_b with dep_c_0_1_2. This is a problem as soon as {dep_a} as well as {dep_b} are exposing types from {dep_c} in their interfaces.

Not supported stuff

Bundled C-code

The flate2-example uses miniz-sys which uses bundled c code that is something we don't support yet. To see the error:

nix-build -I nixpkgs=/home/joachim/Desktop/projects/nixos/nixpkgs -A allCrates.miniz-sys

Tests

Crates come with tests but we didn't have time to implement even though we would have loved to do that. Building crates with tests requires dev-dependencies to be around. We currently just remove them because they caused a lot cyclic dependencies which Nix (and even cargo) can't handle.

Dependencies

Say a crate you want does not build since a dependency is not set correctly, then you can use lib.recursiveUpdate in nix-crates-index/all-carg-packages.nix to change/extend them. This holds also true for pkgs from nixpkgs!

nixcloud / nixcrates