error: path is not valid using nix ==2.18.0
avdv opened this issue · comments
Describe the bug
On CI, we were seeing this problem:
Return code: 1
Error output:
these 2 derivations will be built:
/nix/store/313azxr6w777rrgz0lcgkp5f0dlm2nvr-posix-toolchain.drv
/nix/store/qp53d6mv3gs81akp4gv89pk9k7rb76zc-bazel-nixpkgs-posix-toolchain.drv
this path will be fetched (0.09 MiB download, 0.09 MiB unpacked):
/nix/store/cqam21jchkwyp7k0128g6spcx3mzv6p6-bash-5.1-p16-man
error: path '/nix/store/cqam21jchkwyp7k0128g6spcx3mzv6p6-bash-5.1-p16-man' is not valid
This started to happen around the time when nix 2.18.0 was released. Downgrading to 2.17.0 has resolved this problem.
Steps To Reproduce
- clone the daml project:
git clone https://github.com/digital-asset/daml.git cd damlgit checkout 662a4417912df90ba9d9a99817b8ad9703480d06# not really needed, also fails for commits from 2021- install nix in single-user mode:
bash <( curl -sSfL https://nixos.org/nix/install ) --no-daemon - create
path_invalid.nix:
with import <nixpkgs> { config = {}; overlays = []; };
let
# `packages` might include lists, e.g. `stdenv.initialPath` is a list itself,
# so we need to flatten `packages`.
flatten = builtins.concatMap (x: if builtins.isList x then x else [x]);
env = buildEnv {
name = "posix-toolchain";
paths = flatten [ stdenv.initialPath ];
};
cmd_glob = "${env}/bin/*";
os = if stdenv.isDarwin then "osx" else "linux";
in
runCommand "bazel-nixpkgs-posix-toolchain"
{ executable = false;
# Pointless to do this on a remote machine.
preferLocalBuild = true;
allowSubstitutes = false;
}
''
n=$out/nixpkgs_sh_posix.bzl
mkdir -p "$(dirname "$n")"
cat >>$n <<EOF
load("@rules_sh//sh:posix.bzl", "posix", "sh_posix_toolchain")
discovered = {
EOF
for cmd in ${cmd_glob}; do
if [[ -x $cmd ]]; then
echo " \'$(basename $cmd)\': \'$cmd\'," >>$n
fi
done
cat >>$n <<EOF
}
def create_posix_toolchain():
sh_posix_toolchain(
name = "nixpkgs_sh_posix",
cmds = {
cmd: discovered[cmd]
for cmd in posix.commands
if cmd in discovered
}
)
EOF
''- run:
DADE_NIXPKGS=$(pwd)/nixpkgs-snapshot
if [[ ! -e "$DADE_NIXPKGS" ]]; then
echo "Loading outdated or missing nixpkgs snapshot..."
outpath="$(
nix-store -Q --realise --indirect --add-root "$DADE_NIXPKGS" \
"$(nix-instantiate -Q --eval "nix/nixpkgs.nix" -A path \
| sed 's/^\"//;s/\"$//')"
)"
echo "Done loading the nixpkgs snapshot to $outpath"
fi- run:
NIX_PATH=nixpkgs=$DADE_NIXPKGS nix-build path_invalid.nix
Expected behavior
I would expect the nix-build command to succeed.
nix-env --version output
$ nix-env --version
nix-env (Nix) 2.18.0Additional context
I have tracked down the error message to
Line 727 in 57eb62d
Priorities
Add 👍 to issues you find important.
I'm on commit edolstra@3494c29 in lazy-trees
parent for that merge is edolstra@7f8c99c
from August 16
and I don't have this issue
Should help with bisecting because 2.17 was released on July 24
git log 7f8c99c70c96bf3685e5cad73b38ede801079177^..2.18.0
Also seen in the #nix-dev:nixos.org room
@trofi said
nix-2.18.0 generates very cryptic errors:
$ nix build --no-link -f nixos system --keep-going error: path '/nix/store/0mx9wiw18gn44w97jczfg90s2diwz5zp-jq-1.6-dev' is not validand warnings:
$ nix build --no-link -f nixos system --keep-going warning: output lib of input /nix/store/i7vndr6ilzsjdasjvbd9yjvhll365fbd-dconf-0.40.0.drv missing, aborting the resolving warning: output lib of input /nix/store/i7vndr6ilzsjdasjvbd9yjvhll365fbd-dconf-0.40.0.drv missing, aborting the resolving warning: output info of input /nix/store/0v0vz7p63cx57wzswsf7m79x1yzmmrd8-findutils-4.9.0.drv missing, aborting the resolving
$ nix build -f. jq.devmade it unstick fromerror: path '/nix/store/0mx9wiw18gn44w97jczfg90s2diwz5zp-jq-1.6-dev' is not valid.
#4628 was a big PR so could be that
If anyone is trying to reproduce, I should add to @avdv's description of the bug that if you happen to already have /nix/store/cqam21jchkwyp7k0128g6spcx3mzv6p6-bash-5.1-p16-man in your Nix store this will succeed, which may make reproduction a bit harder.
I can reliably reproduce the error with Nix 2.18.0 on a new Nix installation, though.
@Artturin I'm one of the maintainers of the https://github.com/digital-asset/daml repo and I can reliably reproduce the bug on 2.18.0 and can confirm it does not appear in with 2.17.0. Are there reasonably-easy-to-follow instructions on how to install arbitrary Nix commits instead of released versions? If so I could help with bisecting.
@Artturin spot on! (oh, it's not from PR #4628, but seems related) It is from #8829
5e3986f59cb58f48186a49dcec7aa317b4787522 is the first bad commit
commit 5e3986f59cb58f48186a49dcec7aa317b4787522
Author: John Ericson <John.Ericson@Obsidian.Systems>
Date: Mon Mar 8 16:24:49 2021 -0500
Adapt scheduler to work with dynamic derivations
To avoid dealing with an optional `drvPath` (because we might not know
it yet) everywhere, make an `CreateDerivationAndRealiseGoal`. This goal
just builds/substitutes the derivation file, and then kicks of a build
for that obtained derivation; in other words it does the chaining of
goals when the drv file is missing (as can already be the case) or
computed (new case).
This also means the `getDerivation` state can be removed from
`DerivationGoal`, which makes the `BasicDerivation` / in memory case and
`Derivation` / drv file file case closer together.
The map type is factored out for clarity, and because we will soon hvae
a second use for it (`Derivation` itself).
Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>
.../build/create-derivation-and-realise-goal.cc | 157 +++++++++++++++++++++
.../build/create-derivation-and-realise-goal.hh | 96 +++++++++++++
src/libstore/build/derivation-goal.cc | 22 +--
src/libstore/build/derivation-goal.hh | 15 +-
src/libstore/build/drv-output-substitution-goal.hh | 4 +-
src/libstore/build/entry-points.cc | 11 +-
src/libstore/build/goal.cc | 2 +-
src/libstore/build/goal.hh | 22 ++-
src/libstore/build/substitution-goal.hh | 4 +-
src/libstore/build/worker.cc | 114 ++++++++++++---
src/libstore/build/worker.hh | 22 +++
src/libstore/derived-path-map.cc | 33 +++++
src/libstore/derived-path-map.hh | 73 ++++++++++
tests/dyn-drv/build-built-drv.sh | 4 +-
14 files changed, 523 insertions(+), 56 deletions(-)
create mode 100644 src/libstore/build/create-derivation-and-realise-goal.cc
create mode 100644 src/libstore/build/create-derivation-and-realise-goal.hh
create mode 100644 src/libstore/derived-path-map.cc
create mode 100644 src/libstore/derived-path-map.hh
This started affecting my nixos-unstable-small system flake after a nix-collect-garbage and nix flake update, without anything apparent causing it, with a slightly different derivation:
error: path '/nix/store/phscma235mfx30d1dncgizls1rlmh7iz-bash-5.2-p15-dev' is not valid
I was able to get things back up and running nix-store --repair-path /nix/store/...... on the various broken paths and switching nix.package to be pkgs.nixVersions.nix_2_17.
I can confirm nix-store --repair-path on the path above did work around the issue for me (until next gc?); I didn't need to downgrade to nix_2_17, at least while the path is available.
I can confirm
nix-store --repair-pathon the path above did work around the issue for me (until next gc?); I didn't need to downgrade tonix_2_17, at least while the path is available.
Re the path, are you sure the next one isn't a different bash output? At first I thought it didn't work because there was -info and -man variations.
FWIW, I ran into this with the following store paths:
/nix/store/r0q1ch9gl2f9xkxclg68b7s3qq39lvk1-bash-5.2-p15-man(both systems)/nix/store/af5cs1nfdr1rhghy3vvdmz9jsypgksh7-bash-5.2-p15-doc(both systems)/nix/store/04skl91v03vl3030h3ak3hxafn2v9lzf-kmod-30-dev(.#nixosConfigurations.workyonly)
while upgrading with this commit https://or.computer.surgery/charles/dotfiles/-/commit/7d406812e27e254486ec7a8a01d28bc20eee7694.
My workaround was to run nix copy --from https://cache.nixos.org $affected_store_path for each one until nixos-rebuild worked correctly.
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/amd-rx-7700-xt-not-being-detected-properly/33683/5
It would be handy if someone had a way to reproduce that worked with --store so I didn't have to reinstall Nix, build all of NixOS, etc.
I will work on this as the issue is assigned to me, but posting this here in case someone already has one.
I'm seeing the same problem in CI using 2.18.0 (and not locally, using 2.17.0).
In case it helps, here is a slightly smaller reproduction. Works on a machine with a brand new 2.18.0 installation of Nix, and doesn't rely on the daml repo or Bazel. Still suffers from the issue that I can only reproduce it on brand new installs; once the proper packages have been installed from other sources this succeeds. But on a new install it fails.
EDIT: I can't reproduce this anymore. To avoid confusing people I've commented off the rest of this message.
@garyverhaegen-da Thanks! I am on the Nix team call but I will try this later. In the meantime would you mind trying with --store /some/directory? (that will bypass any daemon / ambient Nix install.)
nix-build repro.nix -A '' -I nixpkgs=include/nixpkgs.nix --store $HOME/repro
using the original (i.e. longest) repro.nix from previous post seems to build successfully. To be extra clear, I am not able to reproduce the issue with --store.
@garyverhaegen-da Oh wow, so when it failed it was also in single user mode global install? It is very odd for --store local (single user mode default) and --store /path/to/dir to have different behavior, as they are almost the same.
Also Could you test with #9081 ? I am not sure that reverting that fixed anything cause I haven't reproduced yet!
This is very weird. I can't reproduce anymore, even on brand new machines. I must have screwed something up. I'll get back to you shortly.
@garyverhaegen-da Other sources of state that perhaps make this non-determinstic is the fetchers catch in ~/.cache.
I must have screwed something up - I'm not able to reproduce using my minimal test case on a brand new machine, though it definitely did work on the machine I was testing with. I'll get back to you as soon as I have a new minimal that works on new machines. And I'll retry --store then.
@garyverhaegen-da No worries, good luck, thank you for doing this it helps me out greatly :)
Alright, I have something again, tried on two new machines so this time I'm reasonably confident. It's way less "minimal" than the previous one though. Here are the steps, on a fresh machine with Nix 2.18.0 and direnv installed:
git clone https://github.com/digital-asset/daml.git(I'd really like to get rid of this but I'm not there yet.)cd damldirenv allow- Add the files below.
- From within the
damlfolder, withdirenvenabled,nix-build repro.nix -A '' -I include/nixpkgs.nix.
Files to add are as follows:
daml/repro.nix
with import <nixpkgs> { config = {}; overlays = []; };
let
flatten = builtins.concatMap (x: if builtins.isList x then x else [x]);
env = buildEnv {
name = "posix-toolchain";
paths = flatten [ stdenv.initialPath ];
};
cmd_glob = "${env}/bin/*";
os = if stdenv.isDarwin then "osx" else "linux";
in
runCommand "bazel-nixpkgs-posix-toolchain"
{ executable = false;
# Pointless to do this on a remote machine.
preferLocalBuild = true;
allowSubstitutes = false;
}
''
n=$out/nixpkgs_sh_posix.bzl
mkdir -p "$(dirname "$n")"
cat >>$n <<EOF
load("@rules_sh//sh:posix.bzl", "posix", "sh_posix_toolchain")
discovered = {
EOF
for cmd in ${cmd_glob}; do
if [[ -x $cmd ]]; then
echo " '$(basename $cmd)': '$cmd'," >>$n
fi
done
cat >>$n <<EOF
}
def create_posix_toolchain():
sh_posix_toolchain(
name = "nixpkgs_sh_posix",
cmds = {
cmd: discovered[cmd]
for cmd in posix.commands
if cmd in discovered
}
)
EOF
''daml/include/nixpkgs.nix
# Pinned version of nixpkgs that we use for our development and deployment.
{ system ? import ./system.nix
, ...
}:
let
# See ./nixpkgs/README.md for upgrade instructions.
src = import ./nixpkgs;
# package overrides
overrides = _: pkgs: rec {
nodejs = pkgs.nodejs-16_x;
nodejs14 = pkgs.nodejs-14_x;
ephemeralpg = pkgs.ephemeralpg.overrideAttrs(oldAttrs: {
installPhase = ''
mkdir -p $out
PREFIX=$out make install
wrapProgram $out/bin/pg_tmp --prefix PATH : ${pkgs.postgresql_11}/bin:$out/bin
'';
});
scala_2_13 = pkgs.scala_2_13.overrideAttrs (oldAttrs: rec {
version = "2.13.10";
name = "scala-2.13.10";
src = pkgs.fetchurl {
url = "https://www.scala-lang.org/files/archive/${name}.tgz";
sha256 = "sha256-sBRhWZzQeGBCxktTN5D0XlG6u5HFLcRl2EaDjpcBnMQ=";
};
});
bazel_5 = pkgs.bazel_5.overrideAttrs(oldAttrs: {
patches = oldAttrs.patches ++ [
# This has been upstreamed but it's only available from Bazel 7.0.0-pre.20230215.2
# (maybe we can get it backported?)
# https://github.com/bazelbuild/bazel/commit/6115d94cd05864fe5c6e5f774e9482b3b4976976
./bazel-retry-cache.patch
];
});
haskell = pkgs.haskell // {
compiler = pkgs.haskell.compiler // {
ghc902 =
if system == "aarch64-darwin" then
pkgs.haskell.compiler.ghc902.override(oldAttrs: {
buildTargetLlvmPackages = pkgs.llvmPackages_12;
llvmPackages = pkgs.llvmPackages_12;
})
else
pkgs.haskell.compiler.ghc902;
};
};
};
nixpkgs = import src {
inherit system;
# pin the overlays
overlays = [overrides];
config.allowUnfree = true;
config.allowBroken = true;
};
in
nixpkgsdaml/include/system.nix
builtins.currentSystem
daml/include/nixpkgs/default.nix
let
spec = builtins.fromJSON (builtins.readFile ./default.src.json);
src = builtins.fetchTarball {
url = "https://github.com/${spec.owner}/${spec.repo}/archive/${spec.rev}.tar.gz";
sha256 = spec.sha256;
};
in
srcdaml/include/nixpkgs/default.src.json
{
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "9a82a9b5248919805a2400266ebd881d5783df2a",
"sha256": "142x1zq3cjadgmvfv0paydlq268pfinllqpq2vl0vxwdiq2nr9iz"
}To make it a bit easier, I've made a branch on the daml repo called debug-nix-2.1.8. I've confirmed the following steps work (i.e. reproduce the bug, i.e. fail) on a new machine with direnv and Nix 2.18.0:
git clone https://github.com/digital-asset/daml.gitcd damlgit checkout 744ef2903053abdbd2e9edf35393c9102f16a439(current tip ofdebug-nix-2.18, adds the five files above)direnv allow- there's unfortunately quite a lot that happens here in the daml reponix-build repro.nix -A '' -I nixpkgs=include/nixpkgs.nix
This is the smallest repro I've got at this point. At least we've eliminated Bazel from the equation. I'll try to get this smaller tomorrow.
If anyone else looks into this in the meantime, I think the most likely variable (compared to running the same repro.nix with the same include folder outside of the daml repo, which works) is the dev-env/etc/nix.conf file, though all of dev-env/ seems plausible.
Note that following these steps but replacing the last step with
nix-build repro.nix -A '' -I nixpkgs=include/nixpkgs.nix --store $HOME/nix-tmp
yields a successful build, i.e. doesn't reproduce the issue. This may point to the core issue being something that happens when the Nix store is initialized in the nix install, or when it is pre-populated by direnv allow. I'm not sure yet how to investigate which of those it might be.
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/2023-09-29-nix-team-meeting-minutes-90/33774/1
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/2023-10-02-nix-team-meeting-minutes-91/33775/1
Also hit this, also for bash, gnugrep and findutils.
Repairing the paths and switching to 2.17 seems to have fixed the issue at this stage of my build.
I can still reliably reproduce on Nix 2.18.0, but I am not able to reproduce on Nix 2.18.1. I'm still working on reducing the repro.
Updating nix in nixpkgs to 2.18.1 NixOS/nixpkgs#258839
@garyverhaegen-da instead of doing --store ..., one can also do export NIX_CONFIG= store = .... This had the advantage of being propogated to all commands (unless something overwrites it), including the direnv one. I hope that would make the custom store version finally repro the issue.
I have to stop for the day, but here's where I'm at: I've managed to completely isolate the problem from the daml repo. Here is a script that reproduces the issue for me.
The ad-hoc.sh command is an internal tool that creates GCP VMs, which do not have Nix installed but do have a writeable-to-all /nix folder, which is why we can do a single-user Nix install with no sudo in the script below.
#!/usr/bin/env bash
set -euo pipefail
DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
cd "$DIR"
./ad-hoc.sh create ubuntu
source .envrc.private
trap "./ad-hoc.sh destroy $MACHINE" EXIT
ssh="ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=$(mktemp)"
while ! $ssh gary@$IP "echo connected" 2>/dev/null; do sleep 1; done
($ssh gary@$IP <<'EOF'
bash <(curl -sSfL https://releases.nixos.org/nix/nix-2.18.0/install)
source $HOME/.nix-profile/etc/profile.d/nix.sh
cat <<'SEMVER' >semver.nix
let
system = builtins.currentSystem;
src = builtins.fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/9a82a9b5248919805a2400266ebd881d5783df2a.tar.gz";
sha256 = "142x1zq3cjadgmvfv0paydlq268pfinllqpq2vl0vxwdiq2nr9iz";
};
pkgs = import src {
inherit system;
config.allowUnfree = true;
config.allowBroken = true;
};
in {
semver = pkgs.callPackage (
{ lib, stdenv, fetchFromGitHub }:
stdenv.mkDerivation rec {
version = "7cd86658";
name = "semver-tool-${version}";
src = fetchFromGitHub {
inherit name;
owner = "fsaintjacques";
repo = "semver-tool";
rev = version;
sha256 = "1v70dgp5yl4di90p8gzbj97zylgc9q971ds5g84id78c2fh3xh28";
};
phases = [ "unpackPhase" "installPhase" ];
installPhase =
''
mkdir -p $out/bin
cp $src/src/semver $out/bin/
patchShebangs $out/bin/semver
'';
meta = {
homepage = https://github.com/fsaintjacques/semver-tool;
description = "The semver shell utility";
license = lib.licenses.gpl3;
platforms = lib.platforms.all;
};
}) {};
}
SEMVER
cat <<'REPRO' > repro.nix
with import <nixpkgs> { config = {}; overlays = []; };
let
flatten = builtins.concatMap (x: if builtins.isList x then x else [x]);
env = buildEnv {
name = "posix-toolchain";
paths = flatten [ stdenv.initialPath ];
};
cmd_glob = "${env}/bin/*";
os = if stdenv.isDarwin then "osx" else "linux";
in
runCommand "bazel-nixpkgs-posix-toolchain"
{ executable = false;
# Pointless to do this on a remote machine.
preferLocalBuild = true;
allowSubstitutes = false;
}
''
n=$out/nixpkgs_sh_posix.bzl
mkdir -p "$(dirname "$n")"
cat >>$n <<EOF
load("@rules_sh//sh:posix.bzl", "posix", "sh_posix_toolchain")
discovered = {
EOF
for cmd in ${cmd_glob}; do
if [[ -x $cmd ]]; then
echo " '$(basename $cmd)': '$cmd'," >>$n
fi
done
cat >>$n <<EOF
}
def create_posix_toolchain():
sh_posix_toolchain(
name = "nixpkgs_sh_posix",
cmds = {
cmd: discovered[cmd]
for cmd in posix.commands
if cmd in discovered
}
)
EOF
''
REPRO
mkdir -p include/nixpkgs
cat <<'NIXPKGS' > include/nixpkgs.nix
# Pinned version of nixpkgs that we use for our development and deployment.
{ system ? import ./system.nix
, ...
}:
let
# See ./nixpkgs/README.md for upgrade instructions.
src = import ./nixpkgs;
# package overrides
overrides = _: pkgs: rec {
nodejs = pkgs.nodejs-16_x;
nodejs14 = pkgs.nodejs-14_x;
ephemeralpg = pkgs.ephemeralpg.overrideAttrs(oldAttrs: {
installPhase = ''
mkdir -p $out
PREFIX=$out make install
wrapProgram $out/bin/pg_tmp --prefix PATH : ${pkgs.postgresql_11}/bin:$out/bin
'';
});
scala_2_13 = pkgs.scala_2_13.overrideAttrs (oldAttrs: rec {
version = "2.13.10";
name = "scala-2.13.10";
src = pkgs.fetchurl {
url = "https://www.scala-lang.org/files/archive/${name}.tgz";
sha256 = "sha256-sBRhWZzQeGBCxktTN5D0XlG6u5HFLcRl2EaDjpcBnMQ=";
};
});
bazel_5 = pkgs.bazel_5.overrideAttrs(oldAttrs: {
patches = oldAttrs.patches ++ [
# This has been upstreamed but it's only available from Bazel 7.0.0-pre.20230215.2
# (maybe we can get it backported?)
# https://github.com/bazelbuild/bazel/commit/6115d94cd05864fe5c6e5f774e9482b3b4976976
./bazel-retry-cache.patch
];
});
haskell = pkgs.haskell // {
compiler = pkgs.haskell.compiler // {
ghc902 =
if system == "aarch64-darwin" then
pkgs.haskell.compiler.ghc902.override(oldAttrs: {
buildTargetLlvmPackages = pkgs.llvmPackages_12;
llvmPackages = pkgs.llvmPackages_12;
})
else
pkgs.haskell.compiler.ghc902;
};
};
};
nixpkgs = import src {
inherit system;
# pin the overlays
overlays = [overrides];
config.allowUnfree = true;
config.allowBroken = true;
};
in
nixpkgs
NIXPKGS
cat <<'SYSTEM' > include/system.nix
builtins.currentSystem
SYSTEM
cat <<'DEFAULT_NIX' > include/nixpkgs/default.nix
let
spec = builtins.fromJSON (builtins.readFile ./default.src.json);
src = builtins.fetchTarball {
url = "https://github.com/${spec.owner}/${spec.repo}/archive/${spec.rev}.tar.gz";
sha256 = spec.sha256;
};
in
src
DEFAULT_NIX
cat <<'DEFAULT_JSON' > include/nixpkgs/default.src.json
{
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "9a82a9b5248919805a2400266ebd881d5783df2a",
"sha256": "142x1zq3cjadgmvfv0paydlq268pfinllqpq2vl0vxwdiq2nr9iz"
}
DEFAULT_JSON
#export NIX_STORE_DIR=$(mktemp -d)
nix-build semver.nix -A "semver"
nix-build repro.nix -A '' -I nixpkgs=include/nixpkgs.nix
EOF
) 2>&1 | grep -oP 'path .* is not valid'I suspect the Nix code can be further reduced/simplified but I'm out of time for today.
Note that uncommenting the line
#export NIX_STORE_DIR=$(mktemp -d)
makes it take a lot longer (a few minutes with the default store, about 2h on these machines with the custom store), and fails to reproduce the issue (i.e. the build succeeds).
Thanks again @garyverhaegen-da for all your hard work.
Note that NIX_STORE_DIR has a different effect (it won't chroot internally so you are still using /nix/store within builds and thus can substitute prebuilt binaries) as opposed to the NIX_CONFIG thing which will do the chroot and thus shouldn't take so long.
@garyverhaegen-da Also, I would be happy to get on a call with you about this, if that might help.
Thanks again @garyverhaegen-da for all your hard work.
Note that
NIX_STORE_DIRhas a different effect (it won't chroot internally so you are still using/nix/storewithin builds and thus can substitute prebuilt binaries) as opposed to theNIX_CONFIGthing which will do the chroot and thus shouldn't take so long.
Can you give me the exact syntax for the NIX_CONFIG option? I'm not very familiar with Nix and the naive export NIX_CONFIG=store=$(mktemp -d) resulted in an error.
I think this is the smallest I can make it:
#!/usr/bin/env bash
set -euo pipefail
DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
cd "$DIR"
./ad-hoc.sh create ubuntu
source .envrc.private
trap "./ad-hoc.sh destroy $MACHINE" EXIT
ssh="ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=$(mktemp)"
while ! $ssh gary@$IP "echo connected" 2>/dev/null; do sleep 1; done
($ssh gary@$IP <<'EOF'
bash <(curl -sSfL https://releases.nixos.org/nix/nix-2.18.0/install)
source $HOME/.nix-profile/etc/profile.d/nix.sh
cat <<'REPRO' > repro.nix
let
system = builtins.currentSystem;
src = builtins.fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/9a82a9b5248919805a2400266ebd881d5783df2a.tar.gz";
sha256 = "142x1zq3cjadgmvfv0paydlq268pfinllqpq2vl0vxwdiq2nr9iz";
};
pkgs = import src {};
flatten = builtins.concatMap (x: if builtins.isList x then x else [x]);
env = pkgs.buildEnv {
name = "posix-toolchain";
paths = flatten [ pkgs.stdenv.initialPath ];
};
cmd_glob = "${env}/bin/*";
in
{
semver = pkgs.callPackage (
{ lib, stdenv, fetchFromGitHub }:
stdenv.mkDerivation rec {
version = "7cd86658";
name = "semver-tool-${version}";
src = fetchFromGitHub {
inherit name;
owner = "fsaintjacques";
repo = "semver-tool";
rev = version;
sha256 = "1v70dgp5yl4di90p8gzbj97zylgc9q971ds5g84id78c2fh3xh28";
};
phases = [ "unpackPhase" "installPhase" ];
installPhase =
''
mkdir -p $out/bin
cp $src/src/semver $out/bin/
patchShebangs $out/bin/semver
'';
}) {};
repro = pkgs.runCommand "repro" {}
''
echo ${cmd_glob}
'';
}
REPRO
nix-build repro.nix -A "semver"
nix-build repro.nix -A 'repro'
EOF
) 2>&1 | grep -oP 'path .* is not valid'At this point everything I can think of either fails in another way or succeeds.
same issue here, I was able to fix it as follows:
❯ sudo nixos-rebuild switch --flake .
warning: Git tree '/home/homeserver/nixos-config' is dirty
building the system configuration...
warning: Git tree '/home/homeserver/nixos-config' is dirty
error: path '/nix/store/wjizx94ksgs39haljg9pn2yinr5ic9az-bash-interactive-5.2-p21-doc' is not valid
❯ nix copy --from https://cache.nixos.org /nix/store/wjizx94ksgs39haljg9pn2yinr5ic9az-bash-interactive-5.2-p21-doc
❯ sudo nixos-rebuild switch --flake .
warning: Git tree '/home/homeserver/nixos-config' is dirty
building the system configuration...
warning: Git tree '/home/homeserver/nixos-config' is dirty
error: path '/nix/store/1bv944c4z7f3d62j462i184aj5lv9nqn-bash-interactive-5.2-p21-info' is not valid
❯ nix copy --from https://cache.nixos.org /nix/store/1bv944c4z7f3d62j462i184aj5lv9nqn-bash-interactive-5.2-p21-info
❯ sudo nixos-rebuild switch --flake .
warning: Git tree '/home/homeserver/nixos-config' is dirty
building the system configuration...
warning: Git tree '/home/homeserver/nixos-config' is dirty
error: path '/nix/store/74jgj1g3sab05jyp7hyppp777cpzjgs3-gnugrep-3.11-info' is not valid
❯ nix copy --from https://cache.nixos.org /nix/store/74jgj1g3sab05jyp7hyppp777cpzjgs3-gnugrep-3.11-info
❯ sudo nixos-rebuild switch --flake .
warning: Git tree '/home/homeserver/nixos-config' is dirty
building the system configuration...
warning: Git tree '/home/homeserver/nixos-config' is dirty
error: path '/nix/store/xg2z6k8zjm2r4xyk8mzycpz09nxv2klc-findutils-4.9.0-info' is not valid
❯ nix copy --from https://cache.nixos.org /nix/store/xg2z6k8zjm2r4xyk8mzycpz09nxv2klc-findutils-4.9.0-info
If you want to
- use 2.18, then please update to Nix greater than or equal to 2.18.1
- or help with troubleshooting dynamic derivations, then a new iteration is available at #9415
I have another reproduction of what I believe to be this issue, on Nix 2.18.1.
In https://github.com/a-h/nix-copy I've got scripts to set up two NixOS VMs - source and target based on the same configuration.nix. The configuration.nix enables SSH, and sets a password for the user adrian of password: https://github.com/a-h/nix-copy/blob/main/configuration.nix
The flake.nix in the repo has a devShell which includes libvirt for running the virtual machines, the xc tool I use to simplify running complex commands, and the virsh-json tool I wrote to parse tables that are output from virsh commands and turn them into JSON.
The first command is xc build-iso which builds an ISO from the configuration.nix.
virt-run-all then runs two copies of the ISO using virt-install. One is called source, and the other is called target.
Once the VMs are stared, the virt-ssh command returns the IP addresses of the source and target machine, then I could run the tests.
Export package to disk
First, I SSH onto the source machine, and copy the hello command to disk:
nix copy --to file://$PWD/hello nixpkgs#hello --extra-experimental-features nix-command --extra-experimental-features flakesCopy from source to target using scp
Next, I copy the results via SCP to the target machine, which mimics the behaviour of using a USB drive etc.
[adrian@nixos:~]$ scp -r ./hello adrian@192.168.122.182:/home/adrian/hello
The authenticity of host '192.168.122.182 (192.168.122.182)' can't be established.
ED25519 key fingerprint is SHA256:O8ltjoGolRWGEFxQV4dK+RktPvTt347gqV8s/QfKA+0.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.122.182' (ED25519) to the list of known hosts.
(adrian@192.168.122.182) Password:
ji01n9vinnj22nbrb86nx8a1ssgpilx8.narinfo 100% 606 581.0KB/s 00:00
i93s7xxblavsacpy82zdbn4kplsyq48l.narinfo 100% 567 5.6MB/s 00:00
9y8pmvk8gdwwznmkzxa6pwyah52xy3nk.narinfo 100% 659 7.2MB/s 00:00
ldrslljw4rg026nw06gyrdwl78k77vyq.narinfo 100% 513 5.6MB/s 00:00
zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm.narinfo 100% 600 6.7MB/s 00:00
12116yvlqh0sszlf6ipagkivxgh4ybn15d7qynvl34k87bmwkyv6.nar.xz 100% 49KB 95.4MB/s 00:00
168qxfl2rswcwbpvajr43r7s2fxan4lk69kyf7wasffga1ava9g8.nar.xz 100% 6357KB 100.9MB/s 00:00
0crc324g4mk9qv2kawr3w0rklvix854m7iarg8z848pibm3sxjfv.nar.xz 100% 87KB 197.6MB/s 00:00
1qi9kgysh7rpfnj20zxiys6xg15265ws0sc5kybmw1zdj8asc37j.nar.xz 100% 50KB 198.3MB/s 00:00
1rm2gqcly43l1mnd7iwyhslcfxnlg91y5c7y49y8dww1qwl0lsyp.nar.xz 100% 402KB 366.1MB/s 00:00
nix-cache-info 100% 21 206.8KB/s 00:00 Get the store path on the source machine
With the data copied to the target machine, I can print out the path. For some reason, there's no newline.
[adrian@nixos:~]$ nix path-info nixpkgs#hello --extra-experimental-features nix-command --extra-experimental-features flakes
evaluating derivation 'flake:nixpkgs#hello'/nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1SSH into target machine and import the results of nix copy
Next, I exit the connection with source and SSH into target.
[adrian@nixos:~]$ exit
logout
Connection to 192.168.122.175 closed.
(nix:nix-shell-env) bash-5.2$ ssh adrian@192.168.122.182
(adrian@192.168.122.182) Password: Then, I try to run the nix copy operation to add the SCP'd files to the store.
[adrian@nixos:~]$ nix copy --all --no-check-sigs --from file://$PWD/hello --extra-experimental-features nix-command --extra-experimental-features flakes
error: path '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4' is not validGot unexpected error
As seen, I got the "error: path '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4' is not valid".
Why does SSH work?
If I log on to source and use ssh as the target, then it works OK:
ssh adrian@192.168.122.175
[adrian@nixos:~]$ nix copy --to ssh-ng://192.168.122.182 nixpkgs#hello --extra-experimental-features nix-command --extra-experimental-features flakes
(adrian@192.168.122.182) Password: Then... on the target...
(nix:nix-shell-env) bash-5.2$ ssh adrian@192.168.122.182
(adrian@192.168.122.182) Password:
Last login: Wed Jan 10 12:57:10 2024 from 192.168.122.1
[adrian@nixos:~]$ nix shell /nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1 --extra-experimental-features nix-command --extra-experimental-features flakes
[adrian@nixos:~]$ hello
Hello, world!So, it's interesting that the nix copy operation seems to work if the target is SSH, but doesn't work if the target is a file path.
Code
I gave @solidsnakedev's workaround a try to help debug the issue further.
I noticed that it is possible to download from the cache to disk, and restore from disk on the target machine. So, I reproduced the issue, having copied the package from source to target. Running on target, I can't add the files to the store:
[adrian@nixos:~]$ nix copy --all --no-check-sigs --from file://$PWD/hello --extra-experimental-features nix-command --extra-experimental-features flakes
error: path '/nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-xgcc-12.3.0-libgcc' is not valid
But if I download it from the Nix cache to disk...
[adrian@nixos:~]$ nix copy --from https://cache.nixos.org /nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-xgcc-12.3.0-libgcc --to file://$PWD/libgcc --extra-experimental-features nix-command Then from disk into the cache...
[adrian@nixos:~]$ nix copy --all --no-check-sigs --from file://$PWD/libgcc --extra-experimental-features nix-command --extra-experimental-features flakesThe operation succeeds:
[adrian@nixos:~]$ nix copy --all --no-check-sigs --from file://$PWD/hello --extra-experimental-features nix-command --extra-experimental-features flakes Was the narinfo or NAR file corrupted during transfer?
My first thought was that perhaps the scp operation didn't work, so I compared the narinfo files, and the one received from the source machine was identical to the one downloaded from the Nix cache on the local machine.
adrian@nixos:~]$ cat hello/ldrslljw4rg026nw06gyrdwl78k77vyq.narinfo
StorePath: /nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-xgcc-12.3.0-libgcc
URL: nar/1qi9kgysh7rpfnj20zxiys6xg15265ws0sc5kybmw1zdj8asc37j.nar.xz
Compression: xz
FileHash: sha256:1qi9kgysh7rpfnj20zxiys6xg15265ws0sc5kybmw1zdj8asc37j
FileSize: 50868
NarHash: sha256:18agdkqz0vzfxznc4h8ss7ql483lyjzjng185xcrwmdx7x1vc9a8
NarSize: 142456
References:
Deriver: m8hpbqa0vsnl1nr6g1n8bg792qakap68-xgcc-12.3.0.drv
Sig: cache.nixos.org-1:v+YiK82YiH+LXgSu9bPFbhPH1KYc190XB7Jb1c8TOvOaN8EsvDsDNcmCOxdLgR6YaB9N0LTp3uIa6ctQrdf/DA==
[adrian@nixos:~]$ cat libgcc/ldrslljw4rg026nw06gyrdwl78k77vyq.narinfo
StorePath: /nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-xgcc-12.3.0-libgcc
URL: nar/1qi9kgysh7rpfnj20zxiys6xg15265ws0sc5kybmw1zdj8asc37j.nar.xz
Compression: xz
FileHash: sha256:1qi9kgysh7rpfnj20zxiys6xg15265ws0sc5kybmw1zdj8asc37j
FileSize: 50868
NarHash: sha256:18agdkqz0vzfxznc4h8ss7ql483lyjzjng185xcrwmdx7x1vc9a8
NarSize: 142456
References:
Deriver: m8hpbqa0vsnl1nr6g1n8bg792qakap68-xgcc-12.3.0.drv
Sig: cache.nixos.org-1:v+YiK82YiH+LXgSu9bPFbhPH1KYc190XB7Jb1c8TOvOaN8EsvDsDNcmCOxdLgR6YaB9N0LTp3uIa6ctQrdf/DA==
In addition, the NAR files themselves are identical.
[adrian@nixos:~]$ sha256sum libgcc/nar/1qi9kgysh7rpfnj20zxiys6xg15265ws0sc5kybmw1zdj8asc37j.nar.xz
f20ca61592ed075e979f8569a07931a284d78df6b17f20a475371fa8fd9b29e2 libgcc/nar/1qi9kgysh7rpfnj20zxiys6xg15265ws0sc5kybmw1zdj8asc37j.nar.xz
[adrian@nixos:~]$ sha256sum hello/nar/1qi9kgysh7rpfnj20zxiys6xg15265ws0sc5kybmw1zdj8asc37j.nar.xz
f20ca61592ed075e979f8569a07931a284d78df6b17f20a475371fa8fd9b29e2 hello/nar/1qi9kgysh7rpfnj20zxiys6xg15265ws0sc5kybmw1zdj8asc37j.nar.xz
I think this shows that the problem doesn't seem to be caused by the download or transfer of the archives.
So what else could be causing this?
Reproducible bisection
Now that I have a reproducible testing process, I noticed that the nixpkgs version was changing between each run, and that I was using a recent commit, so I tried with nixpkgs/23.11 and it worked perfectly well, whereas with the latest unstable nixpkgs it doesn't.
[adrian@nixos:~]$ nix copy --to file://$PWD/hello github:NixOS/nixpkgs/23.11#hello --extra-experimental-features nix-command --extra-experimental-features flakesSince it was working in 23.11, and not in the latest master branch, I decided to hunt it down. I got all the commits since 23.11 on the main branch with git log --oneline | sed '/Release NixOS 23.11/q' and bisected them.
There had been 13,269 commits since the latest, so I had to do 13 checks to find the exact commit when it started to break.
# 76090aacf4b6 - broken
# 7b2399a63c27 - working
# 140f2db977a0 - working
# 468a6bab44bf - broken
# 44f2f5ce5aaf - broken
# f9480bd35d76 - broken
# b122013b2373 - broken
# 0c909de8e6ab - broken
# 5902643e53f2 - broken
# a1e9171ca3c1 - broken
# 73b3a1450f4a - broken
# 67fc0e51da63 - broken
# 5621fb9e2dc5 - working
So I tracked it down to here.
67fc0e51da63 R. Ryantm Fri Dec 15 05:07:22 2023 +0000 biome: 1.4.0 -> 1.4.1
Broken ^
5621fb9e2dc5 Nikolay Korotkiy Tue Dec 12 02:04:18 2023 +0400 mosquitto: fix pkg-config files
Working ^
Why on earth would NixOS/nixpkgs@67fc0e51da63 cause a nix copy operation of hello to print error: path '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4' is not valid?
Reproduction in Nix 2.19.2
Since the workaround was apparently to use Nix 2.18.1, and I was able to reproduce the issue in it, I decided to check if it had been fixed in a subsequent version of Nix accidentally.
I updated the reproduction repo to use 2.19.2 in a-h/nix-copy@e855ecb and then created two new virtual machines - 192.168.122.153 (source) and 192.168.122.152 (target).
0 0 /home/adrian-hesketh/github.com/a-h/nix-copy % ssh adrian@192.168.122.153
The authenticity of host '192.168.122.153 (192.168.122.153)' can't be established.
ED25519 key fingerprint is SHA256:5s8NaYrdeH3MZhUKKrExqw44JmKwS+q9/h8cKkoiFtI.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.122.153' (ED25519) to the list of known hosts.
(adrian@192.168.122.153) Password:
[adrian@nixos:~]$ nix --version
nix (Nix) 2.19.2
[adrian@nixos:~]$ nix copy --to file://$PWD/hello github:NixOS/nixpkgs/67fc0e51da63#hello
[adrian@nixos:~]$ scp -r ./hello/ adrian@192.168.122.152:/home/adrian/hello
The authenticity of host '192.168.122.152 (192.168.122.152)' can't be established.
ED25519 key fingerprint is SHA256:92Q0nHQWTKDy/DdFcCjKgOo8rDSlRNwjAyaoG47a00g.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.122.152' (ED25519) to the list of known hosts.
(adrian@192.168.122.152) Password:
1rm2gqcly43l1mnd7iwyhslcfxnlg91y5c7y49y8dww1qwl0lsyp.nar.xz 100% 402KB 112.7MB/s 00:00
12116yvlqh0sszlf6ipagkivxgh4ybn15d7qynvl34k87bmwkyv6.nar.xz 100% 49KB 179.4MB/s 00:00
0crc324g4mk9qv2kawr3w0rklvix854m7iarg8z848pibm3sxjfv.nar.xz 100% 87KB 170.6MB/s 00:00
168qxfl2rswcwbpvajr43r7s2fxan4lk69kyf7wasffga1ava9g8.nar.xz 100% 6357KB 67.6MB/s 00:00
1qi9kgysh7rpfnj20zxiys6xg15265ws0sc5kybmw1zdj8asc37j.nar.xz 100% 50KB 136.2MB/s 00:00
ldrslljw4rg026nw06gyrdwl78k77vyq.narinfo 100% 513 4.5MB/s 00:00
zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm.narinfo 100% 600 5.6MB/s 00:00
i93s7xxblavsacpy82zdbn4kplsyq48l.narinfo 100% 567 5.6MB/s 00:00
ji01n9vinnj22nbrb86nx8a1ssgpilx8.narinfo 100% 606 5.9MB/s 00:00
nix-cache-info 100% 21 192.7KB/s 00:00
9y8pmvk8gdwwznmkzxa6pwyah52xy3nk.narinfo 100% 659 7.1MB/s 00:00
[adrian@nixos:~]$ exit
logout
Connection to 192.168.122.153 closed.
0 0 /home/adrian-hesketh/github.com/a-h/nix-copy % ssh adrian@192.168.122.152
The authenticity of host '192.168.122.152 (192.168.122.152)' can't be established.
ED25519 key fingerprint is SHA256:92Q0nHQWTKDy/DdFcCjKgOo8rDSlRNwjAyaoG47a00g.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.122.152' (ED25519) to the list of known hosts.
(adrian@192.168.122.152) Password:
[adrian@nixos:~]$ nix copy --all --no-check-sigs --from file://$PWD/hello
error: path '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4' is not valid
[adrian@nixos:~]$ nix --version
nix (Nix) 2.19.2
[adrian@nixos:~]$
@roberth - I would appreciate your thoughts on how to proceed. @'ing you because it says to in your profile. 😁
- I can reproduce the issue on brand new NixOS VMs (see https://github.com/a-h/nix-copy for full code)
- I've tested it using the latest Nix CLI (2.19.2)
- I've hunted it down to a specific
nixpkgscommit.
But the commit doesn't seem to make sense as the cause - I can't see how a minor change to a Rust package would result in error: path '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4' is not valid
Any ideas?
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/tiny-nixos-virtualbox-appliance/38145/3
Reading through the comments above, I noticed that there was a suggestion that it worked in Nix 2.17, but it doesn't.
[adrian@nixos:~]$ nix --version
nix (Nix) 2.17.1
[adrian@nixos:~]$ nix copy --all --verbose --no-check-sigs --from file://$PWD/hello
copying 5 paths...
copying path '/nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-x' from 'file:///home/adrian/hello'...
copying path '/nix/store/i93s7xxblavsacpy82zdbn4kplsyq48l-x' from 'file:///home/adrian/hello'...
copying path '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-x' from 'file:///home/adrian/hello'...
copying path '/nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-x' from 'file:///home/adrian/hello'...
copying path '/nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-x' from 'file:///home/adrian/hello'...
error: path '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4' is not valid
The Nix copy output does contain everything required to restore the store, but it probably isn't doing it in the right order.
The store that's being copied from the source passes verification.
[adrian@nixos:~]$ nix store verify -vvvvv --store "file://$PWD/hello" --all
querying info about '/nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-x' on 'file:///home/adrian/hello'...
querying info about '/nix/store/i93s7xxblavsacpy82zdbn4kplsyq48l-x' on 'file:///home/adrian/hello'...
querying info about '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-x' on 'file:///home/adrian/hello'...
querying info about '/nix/store/kg7q3p9zy4q6spr8a41nd8njni8h8ayg-x' on 'file:///home/adrian/hello'...
querying info about '/nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-x' on 'file:///home/adrian/hello'...
querying info about '/nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-x' on 'file:///home/adrian/hello'...
starting pool of 0 threads
checking '/nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27'...
querying info about '/nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27' on 'file:///home/adrian/hello'...
checking '/nix/store/i93s7xxblavsacpy82zdbn4kplsyq48l-libunistring-1.1'...
querying info about '/nix/store/i93s7xxblavsacpy82zdbn4kplsyq48l-libunistring-1.1' on 'file:///home/adrian/hello'...
checking '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4'...
querying info about '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4' on 'file:///home/adrian/hello'...
checking '/nix/store/kg7q3p9zy4q6spr8a41nd8njni8h8ayg-source'...
querying info about '/nix/store/kg7q3p9zy4q6spr8a41nd8njni8h8ayg-source' on 'file:///home/adrian/hello'...
checking '/nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-xgcc-12.3.0-libgcc'...
querying info about '/nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-xgcc-12.3.0-libgcc' on 'file:///home/adrian/hello'...
checking '/nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1'...
querying info about '/nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1' on 'file:///home/adrian/hello'...
And, the hello check passes:
nix store verify --store "file://$PWD/hello" /nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1
Including recursively.
[adrian@nixos:~]$ nix store verify -vvvvv --recursive --store "file://$PWD/hello" /nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1
querying info about missing paths...
starting pool of 24 threads
querying info about '/nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1' on 'file:///home/adrian/hello'...
querying info about '/nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27' on 'file:///home/adrian/hello'...
querying info about '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4' on 'file:///home/adrian/hello'...
querying info about '/nix/store/i93s7xxblavsacpy82zdbn4kplsyq48l-libunistring-1.1' on 'file:///home/adrian/hello'...
querying info about '/nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-xgcc-12.3.0-libgcc' on 'file:///home/adrian/hello'...
starting pool of 0 threads
checking '/nix/store/i93s7xxblavsacpy82zdbn4kplsyq48l-libunistring-1.1'...
checking '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4'...
checking '/nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-xgcc-12.3.0-libgcc'...
checking '/nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27'...
checking '/nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1'...
Here's the graph of packages produced by nix-store -q --graph /nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1/bin/hello:
From the graph, it's clear that the order of installation must be libunistring-1.1, then libidn2-2.3.4 and xgcc-12.3.0-libgcc can happen at the same time, followed by glibc-2.38-27, and then finally hello-2.12.1.
This is exactly the checking order I see in the logs above. However, nix copy doesn't do the same order.
[adrian@nixos:~]$ nix copy -vvvvv --all --no-check-sigs --from file://$PWD/hello
performing daemon worker op: 31
copying 5 paths...
querying info about '/nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27' on 'file:///home/adrian/hello'...
querying info about '/nix/store/i93s7xxblavsacpy82zdbn4kplsyq48l-libunistring-1.1' on 'file:///home/adrian/hello'...
querying info about '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4' on 'file:///home/adrian/hello'...
querying info about '/nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-xgcc-12.3.0-libgcc' on 'file:///home/adrian/hello'...
querying info about '/nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1' on 'file:///home/adrian/hello'...
copying path '/nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27' from 'file:///home/adrian/hello'...
performing daemon worker op: 44
acquiring write lock on '/nix/var/nix/temproots/3145'
locking path '/nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27'
lock acquired on '/nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27.lock'
copying path '/nix/store/i93s7xxblavsacpy82zdbn4kplsyq48l-libunistring-1.1' from 'file:///home/adrian/hello'...
lock released on '/nix/store/9y8pmvk8gdwwznmkzxa6pwyah52xy3nk-glibc-2.38-27.lock'
copying path '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4' from 'file:///home/adrian/hello'...
copying path '/nix/store/ldrslljw4rg026nw06gyrdwl78k77vyq-xgcc-12.3.0-libgcc' from 'file:///home/adrian/hello'...
copying path '/nix/store/zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1' from 'file:///home/adrian/hello'...
error: path '/nix/store/ji01n9vinnj22nbrb86nx8a1ssgpilx8-libidn2-2.3.4' is not valid
The original logs aren't very helpful, because instead of zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-hello-2.12.1 they have zhl06z4lrfrkw5rp0hnjjfrgsclzvxpm-x, so I did some text edits to the logs above to add the package name instead of -x.
I don't know what it's supposed to be doing, but it looks like it's releasing the glibc lock too early to me. Shouldn't it be holding the lock until all dependencies of glibc (are copied, i.e. libunistring-1.1, libidn2-2.3.4 and xgcc-12.3.0-libgcc)?
Reading through
Lines 1087 to 1169 in e6e160a
Does that help at all?
Since all of the paths are actually available on disk, you can copy them one at a time with a shell script without having to go to the Internet to fetch them again with this (assumes nix-export is the folder containing the results of nix copy to a filesystem):
for x in `grep StorePath nix-export/*.narinfo | awk '{print $2}'`; do nix copy $x --from file://$PWD/nix-export/ --offline; done