Pill 12.5 - GD library not available
jamesdbrock opened this issue · comments
Perhaps I've made a mistake somewhere, but I'm stuck on Pill 12.5. In graphviz.nix I added:
buildInputs = with pkgs; [ gd fontconfig libjpeg bzip2 ];
And then:
$ nix-build graphviz.nix
...
checking for gdlib-config... no
configure: WARNING: GD gdlib-config not found
checking gd.h usability... no
checking gd.h presence... no
checking for gd.h... no
configure: WARNING: Optional GD library not available - no gd.h
...
The nix-build graphviz.nix succeeds, but then it still can't read png.
$ echo 'graph test { a -- b }'|result/bin/dot -Tpng -o test.png
Format: "png" not recognized. ...
Possibly related? libgd/libgd#345
12.4 Digression about gcc and ld wrappers
The gd, jpeg, fontconfig and bzip2 libraries (dependencies of gd) don't use pkg-config to specify which flags to pass to the compiler.
My working theory here is that nix-pills chapter 12 assumes that
The gd, jpeg, fontconfig and bzip2 libraries (dependencies of gd) don't use pkg-config
but in fact libgd has been updated to use pkg-config in the time since this nix-pill was written, and the <nixpkgs> that I'm using in February 2019 contains the new version of libgd, because NixOS/nixpkgs#9682
Maybe we can pin the same version of nixpkgs which the author used to write the pills?
https://nixos.wiki/wiki/FAQ/Pinning_Nixpkgs
In pill 2, we see
2.5. Nixpkgs expressions
More output from the installer:
downloading Nix expressions from `http://releases.nixos.org/nixpkgs/nixpkgs-14.10pre46060.a1a2851/nixexprs.tar.xz'...
but that version of nixpkgs no longer exists...
"No longer exists" - but can be retrieved from Git repos.
Anyway, I think it would be good to update that pill for our days. Now they're official documentation.
What worked for me:
- Checkout the
release-14.12branch of the nixpkgs repository. - replace
binutils-unwrappedwithbinutilsin the autotools.nix - Run
NIX_PATH=nixpkgs=/home/jan/repositories/github.com/NixOS/nixpkgs-release-14.12 nix-build graphviz.nix(replace /home/.../nixpkgs-release-14.12 with your checkout of the release-14.12 branch)- or just replace
import <nixpkgs>withimport /your/nixpkgs/folder/hereingraphviz.nix
- or just replace
I also had a problem with this one. I noticed that the default output of gd didn't include /lib or /include -- I had to map through the buildInputs and append getOutput "lib" / getOutput "dev" to my dependency list that was used to build the CFLAGS / LDFLAGS. I borrowed that strategy from make-derivation.nix.
That got me to the point where graphviz could find the .h files, but it still couldn't find the lib, even though it was in my NIX_LDFLAGS. I chalked that up to being on MacOS, and moved on -- I know nowhere near enough to debug that problem.
I am following nix-pills in nixos-19.03, and by using set -x and echo $NIX_LDFLAGS I can see that setup.sh is finding gd-2.2.5-bin, but not gd-2.2.5 -- both are present in /nix/store, with the latter containing the /lib directory. To me it seems either some change in the bash code is necessary, or I'm only getting gd-bin from buildInputs.
nix-repl> :l <nixpkgs>
Added 10565 variables.
nix-repl> :b gd
this derivation produced the following outputs:
bin -> /nix/store/pp5mb05yy1pkjywmczqjjlpfcdlrpkn9-gd-2.2.5-bin
dev -> /nix/store/5zx8w7pacb8i6q6knqzkbmcjdwpn881z-gd-2.2.5-dev
out -> /nix/store/j54h0xpvkycxl8kk8jn765q1j1pg5qp2-gd-2.2.5
Oh, I guess I know why we can't find the out output:
nix-repl> builtins.toString gd
"/nix/store/pp5mb05yy1pkjywmczqjjlpfcdlrpkn9-gd-2.2.5-bin"
I can't follow and complete the Nix Pills on macOS because of this issue :(
Has someone found a solution yet? Thank you..
Did you try the workaround that I posted?
@jheidbrink thx for providing this workaround. This worked for me as well. The tutorial really should be fixed though.
@jheidbrink I have to add something on top of your workaround so section 12.6 will also work.
The problem is the change of binutils vs. binutils-unwrapped.
This will make nix-build -A hello fail (due to error 127 libhello.a and ar command not found)
The solution(my solution) is to add both binutils and binutils-unwrapped to the buildInputs of autotools.nix
and then change graphviz.nix to :
let
pkgs = import /home/jane/nixpkgs {};
patchedPkgs = pkgs // {"binutils-unwrapped"= "";};
mkDerivation = import ./autotools.nix patchedPkgs;
in mkDerivation {
name = "graphviz";
src = ./graphviz-2.38.0.tar.gz;
buildInputs = with pkgs; [ gd fontconfig libjpeg bzip2 ];
}
And finally for the single respository expression (Nix Pill 12.7) , i made this: https://gist.github.com/ThoenigAdrian/c41dea2ff44e3b15a97725cd2830b36b
Wasn't Nix supposed to be reproducible? This thread seems to prove that Nix is not even close to being reproducible.
Can someone explain this? What exactly are the limits to Nix's reproducibility?
Hey @hydramo , as I am still learning nix i may be the wrong one to answer but i'll try anyways :) .
Imo this tutorial should be written differently, as to first "git clone " the nix respository then tell the user to checkout a specific commit and then everything should be built on this specific commit.
- This would make the entire tutorial reproducible
- It wouldn't need to be maintained when packages change (see binutils vs binutils-unwrapped issue)
- It would provide some insight that you can check out the repository and use a specific commit. In order to pinn packages.
So as far as i understood it. A nix build is only reproducible if you use the same git commit of the packages repository.
to the original post, the configure step needs to pick up libpng for it to compile and link against it to enable the ability to read png's. This is not present in the nix-pills example, but it's also not meant to serve as an exhaustive way to build graphviz. If you want read to png's, libpng needs to be added to buildInputs.
Wasn't Nix supposed to be reproducible? This thread seems to prove that Nix is not even close to being reproducible.
Can someone explain this? What exactly are the limits to Nix's reproducibility?
There's limits to getting binary reproducibility as someone can find a source of randomness (e.g. /dev/random), so side-effectful things can cause non-determinism. nix-build has a --check option will attempt to rebuild the derivation and verify that the package produced the same build outputs.
Also this determinism is dependent on what's included in the expression, the line pkgs = import <nixpkgs> { }; is importing a completely different from the original, and it's understandable that the packages and tooling have changed with time. If you want extreme reproducibility then you need to pin nixpkgs such as https://nixos.wiki/wiki/FAQ/Pinning_Nixpkgs. That way you're guaranteed to get the same bits.
The 12.5 nix-pills is meant to demonstrate a bootstrapped build environment and builds upon the previous pills. To make the expression as exhaustive as the expression found on nixpkgs wouldn't fit the "pill" intent. https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/graphics/graphviz/base.nix
I bumped into the same issue. I have to admit: it is weird that a tutorial linked from the official homepage cannot be completed without checking github issues.
@jheidbrink I tried your solution and was still unable to compile graphviz with PNG support. I didn't actually see a Release 14.12 tag on nixpkgs so I went with the 0.14 tag.
I first tried setting the NIX_PATH variable and hard coding the path into the nix expression.
This issue is somewhat twofold. Firstly, these pills where written against version 14.x of <nixpkgs>. That explains why it fails now. Secondly, in the port to this repo, there have been some modifications so that they might work against current versions of <nixpkgs> — notably, replacing binutils with binutils-unwrapped. That explains why the previous pills have been working so far. The porting/modification, however, never reached building graphviz with png support. Section 12.7 still links to the original gist.
So here is pinning Nixpkgs to version 14.12 for section 12.5.
./graphviz.nix
let
# pkgs = import <nixpkgs> {};
pkgs = import (builtins.fetchGit {
name = "nixpkgs-14.12";
# $ git ls-remote https://github.com/nixos/nixpkgs | grep release-14.12
# 770822493e3962d795739fafd522e771bfe06e3d refs/heads/release-14.12
# 770822493e3962d795739fafd522e771bfe06e3d refs/remotes/origin/release-14.12
url = "https://github.com/nixos/nixpkgs";
ref = "refs/heads/release-14.12";
rev = "770822493e3962d795739fafd522e771bfe06e3d";
}) { };
mkDerivation = import ./autotools.nix pkgs;
src = builtins.fetchurl {
# $ nix-prefetch-url http://pkgs.fedoraproject.org/repo/pkgs/graphviz/graphviz-2.38.0.tar.gz/5b6a829b2ac94efcd5fa3c223ed6d3ae/graphviz-2.38.0.tar.gz
# [24.7 MiB DL]
# path is '/nix/store/s92nsvg5slyrwiy3gw6wl05m6dbx9ym7-graphviz-2.38.0.tar.gz'
# 17l5czpvv5ilmg17frg0w4qwf89jzh2aglm9fgx0l0aakn6j7al1
url =
"http://pkgs.fedoraproject.org/repo/pkgs/graphviz/graphviz-2.38.0.tar.gz/5b6a829b2ac94efcd5fa3c223ed6d3ae/graphviz-2.38.0.tar.gz";
sha256 = "17l5czpvv5ilmg17frg0w4qwf89jzh2aglm9fgx0l0aakn6j7al1";
};
in mkDerivation {
name = "graphviz";
inherit src;
buildInputs = with pkgs; [ gd fontconfig libjpeg bzip2 ];
}./autotools.nix
pkgs: attrs:
let
defaultAttrs = with pkgs; {
builder = "${bash}/bin/bash";
args = [ ./builder.sh ];
setup = ./setup.sh;
baseInputs = [
# binutils-unwrapped – was not yet added in Nixpkgs version 14.x
binutils
coreutils
findutils
gawk
gcc
gnugrep
gnumake
gnused
gnutar
gzip
patchelf
];
buildInputs = [ ];
system = builtins.currentSystem;
};
in derivation (defaultAttrs // attrs)And this is ./test.png
Thanks a lot @gomain! Your post allowed me to reproduce the example.
As a general feedback, I would prefer updating the pill so that it works with the current version of nixpkgs since my intent in learning nix is to work with it in its current state and to evolve along with it. Being able to reproduce an example with a previous version is useful and much better than not being able to reproduce the example at all. Even better would be to be able to reproduce the example with the current or a recent version of nixpkgs.
The pills could indeed be versioned in such a way that it is made clear with which version of nixpkgs they were produced and henceforth offer no guarantee of working with other versions.
As a newcomer to the nix community I would be happy to help with updating the documentation so that it works with a recent version.
So does anyone know what is missing to make the example work with the current nixpkgs version? The comment #107 (comment) by @jamesdbrock seems to provide some hints ...
I feel graviz has grown to be too complex for use in these Pills. Such updating would probably be replacing it with another package that demonstrates this same idea: configuring a build by selecting its dependencies.
I'm wondering whether this custom autotools.nix is powerful enough to demonstrate the configuring a build by selecting its dependencies idea without requiring substantial extra code. As an alternative, we could drop that demonstration and let the chapter focus on its title: The inputs pattern.
To demonstrate the inputs pattern, all we want is to have a second derivation that uses autotools.nix. I found that GNU Grep works well with autotools.nix without further adjustments. I'd suggest to use that instead of Graphviz as a quick way to fix this issue. More complex examples could always be added later on, and possibly in a later chapter, for example after the stdenv got introduced.
edit: This might not be such a good idea. After all, the following chapters use the graphviz derivation as motivation for the callPackage and override design patterns.
@jheidbrink maybe both. A small example using grep in order to show the concept, and graphviz as a more elaborated example.
I just got bit by this same thing, working through Nix Pills.
I understand the arguments being raised here, and agree that probably graphviz as it exists today is not the right example. However, I think that having something that works is the most important thing, as working through the tutorial and having something fail, and having to come here to find the solution lowers confidence in the nix process.
Fixing this enough so that things work as expected on the current release (20.09) should be priority 1, then figuring out some alternative that is simpler can follow.
cc @garbas
I just had a similar experience. It was frustrating.
Same here, got stuck, then found this issue.
I had the same issue. For reference, I did try to create a gd derivation from scratch using the current 2.3 source code. That worked successfully, and I was able to import it into the graphiz derivation successfully (and I confirmed it contained the header files). However, it still did NOT work with the mkDerivation function we have been using so far (still said the gd headers couldn't be found).
Fairly sure the issue is with the simplicity of our autotools.nix and setup.sh. Would strongly recommend that we just drop the png example as the quick fix as it really wasn't important for understanding the core concepts. In the long-run, I definitely recommend switching out graphiz.
Just got bit by this too. :(
Can we please fix the pill?
What we really need to fix this pill?
Please increment the counter, I am another victim :)
My understanding of how to fix it is explained below. These are just thoughts from a noob but with a steer from legitimate community members, I feel I could help with a 2- maybe 3-star solution which should still be better than 1-star nothing in the short term.
1-star solution: noop
Just don't anything but let's not fool ourselves, this is pretty much the gate to the community for new-joiners and it's broken. To one familiar with the wisdom of the crowds and Social Architecture, red alerts should fire off all over right now in the nix community.
2-star solution: warn
Add a link from that section to this issue: Currently broken: check #107
3-star solution: fix
Apply @gomain's fix straight without much explanation.
4-star solution: learn from it
- 3-star solution
- Add a chapter/section earlier on pinning nixpkgs and expanding on requirements that support reproducibility. Once I found this issue, I was actually happy with all I learnt.
5 star solution: prevent
- 4-star solution
- add a link on every pill to this repo, possibly in the footer: "having trouble? check issues or create one here"
this is pretty much the gate to the community for new-joiners
I don't really want to hijack this thread, but I'm not sure if the pills should be the "gate for new-joiners": they go into quite some implementation details and non-typical usage that doesn't seem very important to understand in your first steps (also discussed in https://discourse.nixos.org/t/new-users-using-nixos-wrong/9973/20 ) - I think it's good https://nixos.org/learn.html doesn't show it very prominently anymore.
That said, it's still a cool resource when you're diving in deeper, and it'd be good to keep them accurate, so thanks a lot for pushing towards fixing!
@raboof Picked up Nix and NixOS as my daily drivers earlier this year. Before I made the switch, I stumbled around in the dark, got stuck in a lot of areas, and almost threw in the towel. Then I found the pills and the step-by-step build-up made everything start to click. None of the other resources on https://nixos.org/learn.html start from first principles of nix which makes it confusing given the highly opinionated nature of the ecosystem.
Overall, I thought the pills were a great intro (minus the fact they are a bit outdated). I'd recommend the pills get some more love rather than throwing the baby out with the bath water.
this is pretty much the gate to the community for new-joiners
I'm not sure if the pills should be the "gate for new-joiners"
There must be some sort of ladder between "look, it is all magic" and nixpkgs, right?
Then I found the pills and the step-by-step build-up made everything start to click
Interesting, I had the opposite experience: I started with the pills and was ready to give up, then started to understand things as I got familiar with the manuals. I guess it's a combination of the fact that different people learn in different ways, and perhaps our "stumbling" did give us the background needed to see the whole picture.
I'd recommend the pills get some more love rather than throwing the baby out with the bath water.
There must be some sort of ladder between "look, it is all magic" and nixpkgs, right?
(I definitely agree, we should improve them rather than dropping them entirely)
Please increment the counter, I am another victim :)
- 1
2-star solution: warn
Add a link from that section to this issue: Currently broken: check #107
I agree a warning would already be super helpful here.
Nix pills is the gate to the community as it was the first introduction I could find on how to write a derivation that I was able to follow. It is also mentioned at the top on the wiki on how to start packaging.
The other introduction in the manual called Part IV. Writing Nix Expressions can't work for someone new to Nix, as it requires editing all-packages.nix without explanation where that file lives.
Does it be "Nix way" to just "pin" every script in pills to version 14.12? Those we "reuse" Chapter of pinning, and idea of "reproductive build".
Hi there, i just had a look into this.
From my observation, the configure script tries to compile a source file like #include <gd.h> ... and does not succeed.
So i built my own file like this:
#include <gd.h>
int main() { puts("bla"); }and tried to compile it like this:
[tfc@jongepad:/tmp/pills]$ nix-shell graphviz.nix
[nix-shell:/tmp/pills]$ source $setup
[nix-shell:/tmp/pills]$ echo $NIX_CFLAGS_COMPILE
-I /nix/store/p2l3yfj0fd7d0q39kspsc959z56dfq3v-gd-2.3.0-dev/include -I /nix/store/4xkhd51k7y6g9vi2h9kfvcxd67k3v3f1-gawk-5.1.0/include -I /nix/store/5ddb4j8z84p6sjphr0kh6cbq5jd12ncs-binutils-2.35.1/include -I /nix/store/wfzdk9vxayfnw7fqy05s7mmypg5a8lyr-gnumake-4.3/include
[nix-shell:/tmp/pills]$ NIX_DEBUG=1 gcc bla.c
HARDENING: disabled flags: fortify stackprotector strictoverflow format pic pie
extra flags before to /nix/store/sjhz1j2d1ssn59f66kqp92xj9mpsww2d-gcc-10.3.0/bin/gcc:
-Wl\,-dynamic-linker=/nix/store/jsp3h3wpzc842j0rz61m5ly71ak6qgdn-glibc-2.32-54/lib/ld-linux-x86-64.so.2
original flags to /nix/store/sjhz1j2d1ssn59f66kqp92xj9mpsww2d-gcc-10.3.0/bin/gcc:
bla.c
extra flags after to /nix/store/sjhz1j2d1ssn59f66kqp92xj9mpsww2d-gcc-10.3.0/bin/gcc:
-B/nix/store/50msfhkz5wbyk8i78pjv3y9lxdrp7dlm-gcc-10.3.0-lib/lib
-B/nix/store/jsp3h3wpzc842j0rz61m5ly71ak6qgdn-glibc-2.32-54/lib/
-idirafter
/nix/store/q141hd8jl7in5223jmf7kmx9h517km4p-glibc-2.32-54-dev/include
-idirafter
/nix/store/sjhz1j2d1ssn59f66kqp92xj9mpsww2d-gcc-10.3.0/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/include-fixed
-B/nix/store/88ghxafjpqp5sqpd75r51qqg4q5d95ss-gcc-wrapper-10.3.0/bin/
-L/nix/store/jsp3h3wpzc842j0rz61m5ly71ak6qgdn-glibc-2.32-54/lib
-L/nix/store/50msfhkz5wbyk8i78pjv3y9lxdrp7dlm-gcc-10.3.0-lib/lib
bla.c:1:10: fatal error: gd.h: No such file or directory
1 | #include <gd.h>
| ^~~~~~
compilation terminated.when looking into the wrapper script:
[nix-shell:/tmp/pills]$ grep NIX_CFLAGS /nix/store/88ghxafjpqp5sqpd75r51qqg4q5d95ss-gcc-wrapper-10.3.0/bin/gcc
NIX_CFLAGS_COMPILE_x86_64_unknown_linux_gnu+=" $NIX_CXXSTDLIB_COMPILE_x86_64_unknown_linux_gnu"
NIX_CFLAGS_LINK_x86_64_unknown_linux_gnu+=" $NIX_CXXSTDLIB_LINK_x86_64_unknown_linux_gnu"
extraAfter=($NIX_CFLAGS_COMPILE_x86_64_unknown_linux_gnu)
extraBefore=(${hardeningCFlags[@]+"${hardeningCFlags[@]}"} $NIX_CFLAGS_COMPILE_BEFORE_x86_64_unknown_linux_gnu)
extraAfter+=($NIX_CFLAGS_LINK_x86_64_unknown_linux_gnu)it seems like the wrapper does not even look for NIX_CFLAGS_COMPILE but only for NIX_CFLAGS_COMPILE_x86_64_unknown_linux_gnu.
It seems to me like this is more the error than graphviz being too complicated for this example nowadays.
This smells like some cross-related changes in our gcc wrapper. @Ericson2314, does that ring any bells?