R libs not found at runtime

Question

R libs not found at runtime

eltix opened this issue 4 years ago · comments

Eliott Tixier commented 4 years ago

Bug description

R external dependencies (jsonlite in the example below) are not found at runtime.

To reproduce

Below is a minimal example to reproduce the bug.

Sources

tree

├── factorial-design.cabal
├── Main.hs
├── main.R
├── Setup.hs
└── shell.nix

Main.hs

{-# LANGUAGE QuasiQuotes #-}
{-# OPTIONS_GHC -w #-}
module Main where

import           Foreign.R           (SEXP, SEXPTYPE)
import qualified Foreign.R           as R
import           Language.R.Instance as R
import           Language.R.QQ

wrapR :: R s ()
wrapR = do
  [r|
    library("jsonlite")
    print(s_hs)
  |]
  return ()
  where
    s = "Hello, World!"

main :: IO ()
main = do
  R.withEmbeddedR R.defaultConfig $ R.runRegion $ wrapR

shell.nix

let
  nixpkgsRev = "bc260badaebf67442befe20fb443034d3a91f2b3"; # 20.09-beta
  nixpkgsSha256 = "1iysc4xyk88ngkfb403xfq5bs3zy29zfs83pn99kchxd45nbpb5q";

  nixpkgs = fetchTarball {
    url = "https://github.com/nixos/nixpkgs/archive/${nixpkgsRev}.tar.gz";
    sha256 = nixpkgsSha256;
  };

  pkgs = import nixpkgs {};
  inherit (pkgs) haskellPackages;

  haskellDeps =  ps: with ps; [
    base
    inline-r
  ];
  ghc = haskellPackages.ghcWithPackages haskellDeps;

  stdenv = pkgs.stdenv;
  renv = pkgs.rWrapper.override{ packages = with pkgs.rPackages; [
      jsonlite
      R_utils
      ];};
in with stdenv; {
  myProject = stdenv.mkDerivation {
    name = "factorial-design";
    version = "1";
    src = if pkgs.lib.inNixShell then null else nix;
    isLibrary = false;
    isExecutable = true;
    buildInputs = [
      pkgs.glibcLocales
      renv
      haskellPackages.cabal-install
      ghc
    ];
    # workaround suggested in https://github.com/tweag/HaskellR/issues/346 that doesn't solve the pb
    executableSystemDepends = [
      renv
    ];
  };
}

main.R

library("jsonlite")

boo = all.equal(mtcars, fromJSON(toJSON(mtcars)))
print(boo)

factorial-design.cabal (excerpt)

executable factorial-design
  main-is:             Main.hs
  -- other-modules:
  -- other-extensions:
  build-depends:       base >=4.13 && <4.14
                               , inline-r

Commands to run

(in a pure nix-shell)

> cabal build 
> cabal run

throws the following error:

factorial-design: R Runtime Error: Error in library("jsonlite") : there is no package called 'jsonlite

However, running the R script directly with

Rscript main.R

works fine.

Expected behavior

It is expected that package 'jsonlite' be found

Environment

OS name + version: Ubuntu, 20.04.1 LTS (Focal Fossa)
nix config

 - system: `"x86_64-linux"`
 - host os: `Linux 5.4.0-48-generic, Ubuntu, 20.04.1 LTS (Focal Fossa)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.7`
 - channels(root): `"nixpkgs-19.09pre190327.54f385241e6"`
 - channels(etixier): `"unstable-20.09pre236721.840c782d507"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

Additional context
Existing issues related to the problem: #287 #328 #344 #346
The proposed solutions all revolve around adding R (or the R environment) to the executableSystemDepends attribute of mkDerivation which doesn't work in my case.

There's also a dubious solution involving hardcoding the R libs paths in the shell.nix.

Notify

@YPares

idontgetoutmuch · Answer 1 · Fri Oct 02 2020 22:15:21 GMT+0800 (China Standard Time)

Would setting R_LIBS_SITE help?

Eliott Tixier · Answer 2 · Fri Oct 02 2020 22:17:40 GMT+0800 (China Standard Time)

@idontgetoutmuch Could you please elaborate? What should R_LIBS_SITE be set too? Also, is this the expected behavior? I was expecting this to work out of the box.

idontgetoutmuch · Answer 3 · Fri Oct 02 2020 22:27:28 GMT+0800 (China Standard Time)

What does .libPaths() say for both invocations?

idontgetoutmuch · Answer 4 · Fri Oct 02 2020 22:42:01 GMT+0800 (China Standard Time)

I tried your shell.nix but got a different error :(

Preprocessing library for inline-r-0.10.3..
running dist/build/Foreign/R_hsc_make failed (exit code -6)
rsp file was: "dist/build/Foreign/hsc2hscall28974-3.rsp"
output file:"dist/build/Foreign/R.hs"
command was: dist/build/Foreign/R_hsc_make  >dist/build/Foreign/R.hs
error: dyld: Library not loaded: libRblas.dylib
  Referenced from: /nix/store/yq3k8r3amdkva5xjjgapl1nn1vdrnqkh-R-4.0.2/lib/R/lib/libR.dylib
  Reason: image not found

builder for '/nix/store/fjwpcn11faz8jn368cw0iz4k9nxqfnrc-inline-r-0.10.3.drv' failed with exit code 1
cannot build derivation '/nix/store/1i7pq04wlyzpj4mnyqg7aii1xzj6vl4g-ghc-8.8.4-with-packages.drv': 1 dependencies couldn't be built
error: build of '/nix/store/1i7pq04wlyzpj4mnyqg7aii1xzj6vl4g-ghc-8.8.4-with-packages.drv', '/nix/store/zzp6hx7ghflhyqhplsaj0x4dh55rw3x6-R-4.0.2-wrapper.drv' failed

idontgetoutmuch · Answer 5 · Fri Oct 02 2020 22:42:57 GMT+0800 (China Standard Time)

I probably need to make the dependency on BLAS explicit somewhere but I am not going to have time to do that today.

idontgetoutmuch · Answer 6 · Fri Oct 02 2020 22:59:39 GMT+0800 (China Standard Time)

This may be useful for me NixOS/nixpkgs#34209 but probably not for @eltix.

idontgetoutmuch · Answer 7 · Sun Oct 04 2020 02:13:53 GMT+0800 (China Standard Time)

@eltix I don't even get as far as you

\[\][nix-shell:~/eltix]$\[\] cabal build
Resolving dependencies...
Build profile: -w ghc-8.8.4 -O1
In order, the following will be built (use -v for more details):
 - eltix-0.1.0.0 (exe:eltix) (first run)
Configuring executable 'eltix' for eltix-0.1.0.0..
Warning: The 'license-file' field refers to the file 'LICENSE' which does not
exist.
Preprocessing executable 'eltix' for eltix-0.1.0.0..
Building executable 'eltix' for eltix-0.1.0.0..
[1 of 1] Compiling Main             ( Main.hs, /Users/dom/eltix/dist-newstyle/build/x86_64-osx/ghc-8.8.4/eltix-0.1.0.0/x/eltix/build/eltix/eltix-tmp/Main.o )
Error: package or namespace load failed for 'stats' in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/nix/store/kr0rkih4gfi09f08w69dzh2iga41x0bj-R-4.0.2/lib/R/library/stats/libs/stats.so':
  dlopen(/nix/store/kr0rkih4gfi09f08w69dzh2iga41x0bj-R-4.0.2/lib/R/library/stats/libs/stats.so, 6): Library not loaded: libRlapack.dylib
  Referenced from: /nix/store/kr0rkih4gfi09f08w69dzh2iga41x0bj-R-4.0.2/lib/R/library/stats/libs/stats.so
  Reason: image not found
During startup - Warning message:
package 'stats' in options("defaultPackages") was not found 
Linking /Users/dom/eltix/dist-newstyle/build/x86_64-osx/ghc-8.8.4/eltix-0.1.0.0/x/eltix/build/eltix/eltix ...

idontgetoutmuch · Answer 8 · Sun Oct 04 2020 02:15:37 GMT+0800 (China Standard Time)

BTW I had to patch default.nix for R to get this far but this is a macos issue so shouldn't affect you.

idontgetoutmuch · Answer 9 · Sun Oct 04 2020 16:46:50 GMT+0800 (China Standard Time)

By going back to 20.03 I have avoided the error about package 'stats' not being found. This works for me

let

nixpkgs = fetchTarball {
  url = "https://github.com/NixOS/nixpkgs-channels/archive/nixos-20.03.tar.gz";
  sha256 = "1qbs7p0mmcmpg70ibd437hl57byqx5q0pc61p1dckrkazj7kq0pc";
};

# nixpkgs = builtins.fetchGit {
#   url = "file:///Users/dom/nixpkgs";
#   rev = "8cb04412db02d6be077e74f2c0caca43cbc20977";
#   };

pkgs = import nixpkgs {};
inherit (pkgs) haskellPackages;

haskellDeps = ps: with ps; [
  base
  (pkgs.haskell.lib.dontCheck inline-r)
];

ghc = haskellPackages.ghcWithPackages haskellDeps;

nixPackages = [
  ghc
  pkgs.R
  pkgs.rPackages.jsonlite
  ];

in

pkgs.mkShell {
  buildInputs = [
    nixPackages
  ];
}

idontgetoutmuch · Answer 10 · Sun Oct 04 2020 16:53:36 GMT+0800 (China Standard Time)

\[\][nix-shell:~/eltix]$\[\] cabal run
Up to date
[1] "Hello, World!"

I can't tell you why the two nix derivations give different results.

Eliott Tixier · Answer 11 · Mon Oct 05 2020 15:24:51 GMT+0800 (China Standard Time)

Thanks Dominic, I'll try your suggestion

Eliott Tixier · Answer 12 · Mon Oct 05 2020 17:43:22 GMT+0800 (China Standard Time)

This indeed works for me too:

let
  nixpkgsRev = "bc260badaebf67442befe20fb443034d3a91f2b3"; # 20.09-beta
  nixpkgsSha256 = "1iysc4xyk88ngkfb403xfq5bs3zy29zfs83pn99kchxd45nbpb5q";

  nixpkgs = fetchTarball {
    url = "https://github.com/nixos/nixpkgs/archive/${nixpkgsRev}.tar.gz";
    sha256 = nixpkgsSha256;
  };

  pkgs = import nixpkgs {};
  inherit (pkgs) haskellPackages;

  haskellDeps = ps: with ps; [
    base
    (pkgs.haskell.lib.dontCheck inline-r)
  ];

  ghc = haskellPackages.ghcWithPackages haskellDeps;

  nixPackages = [
    ghc
    pkgs.R
    pkgs.rPackages.jsonlite
    ];

  in

  pkgs.mkShell {
    buildInputs = [
      nixPackages
      pkgs.cabal-install
    ];
  }

@YPares @teto

Eliott Tixier · Answer 13 · Mon Oct 05 2020 19:15:18 GMT+0800 (China Standard Time)

@idontgetoutmuch should I close this issue then? The proposed solution doesn't follow the guidelines in https://nixos.wiki/wiki/R where they recommend using rWrapper.

idontgetoutmuch · Answer 14 · Mon Oct 05 2020 19:27:43 GMT+0800 (China Standard Time)

I think close it and I will raise it on the nixpkgs issue tracker.

idontgetoutmuch · Answer 15 · Mon Oct 05 2020 19:32:02 GMT+0800 (China Standard Time)

@eltix BTW what platform are you on?

Eliott Tixier · Answer 16 · Mon Oct 05 2020 20:11:39 GMT+0800 (China Standard Time)

@idontgetoutmuch Ubuntu 20.04
Thanks again for your help.

idontgetoutmuch · Answer 17 · Sat Feb 19 2022 23:27:14 GMT+0800 (China Standard Time)

As a coda to this and because I now rWrapper which is the correct way, I am leaving my current shell.nix below

let

  pkgs = builtins.fetchTarball {
    # url = "https://github.com/NixOS/nixpkgs/archive/21.11.tar.gz";
    # sha256 = "162dywda2dvfj1248afxc45kcrg83appjd0nmdb541hl7rnncf02";
    url = "https://github.com/NixOS/nixpkgs/archive/nixpkgs-unstable.tar.gz";
    sha256 = "0v3c4r8v40jimicdxqvxnzmdypnafm2baam7z131zk6ljhb8jpg9";
};

myHaskellPackageOverlay = self: super: {
  myHaskellPackages = super.haskell.packages.ghc921.override {
    overrides = hself: hsuper: rec {

      hmatrix-sundials = super.haskell.lib.dontCheck (
        hself.callCabal2nix "hmatrix-sundials" (builtins.fetchGit {
          url = "https://github.com/novadiscovery/hmatrix-sundials";
          rev = "76bfee5b5a8377dc3f7161514761946a60d4834a";
        }) { sundials_arkode          = self.sundials;
             sundials_cvode           = self.sundials;
             klu                      = self.suitesparse;
             suitesparseconfig        = self.suitesparse;
             sundials_sunlinsolklu    = self.sundials;
             sundials_sunmatrixsparse = self.sundials;
           });

      inline-r = super.haskell.lib.dontCheck (
        hself.callCabal2nixWithOptions "inline-r" (builtins.fetchGit {
          url = "https://github.com/tweag/HaskellR";
          rev = "c3ba1023480e26ade420896bcb629ceaad59f308";
        }) "--subpath inline-r" { });

      singletons = self.haskell.lib.addBuildDepends(
        hself.callHackageDirect {
          pkg = "singletons";
          ver = "3.0.1";
          sha256 = "sha256-ixHWZae6AxjRUldMgpYolXBGsOUT5ZVIw9HZkxrhHQ0=";
        } { }) [ ];

      random-fu = super.haskell.lib.dontCheck (
        hself.callCabal2nixWithOptions "random-fu" (builtins.fetchGit {
          url = "https://github.com/haskell-numerics/random-fu";
          rev = "18a6ba6b29c7ca3b3ff34ea6ca0eca910da72726";
        }) "--subpath random-fu" { });

      rvar = super.haskell.lib.dontCheck (
        hself.callCabal2nixWithOptions "rvar" (builtins.fetchGit {
          url = "https://github.com/haskell-numerics/random-fu";
          rev = "18a6ba6b29c7ca3b3ff34ea6ca0eca910da72726";
        }) "--subpath rvar" { });

    };
  };
};

in

{ nixpkgs ? import pkgs {
  config.allowBroken = true;
  config.allowUnsupportedSystem = true;
  overlays = [ myHaskellPackageOverlay ]; }
}:

let

  pkgs = nixpkgs;

  R-with-my-packages = pkgs.rWrapper.override{
    packages = with pkgs.rPackages; [
      # JuliaCall
      # reticulate
      # tidyverse
      # rmarkdown
      # Rcpp
      # RcppEigen
      # tidyverse
      # doParallel
      # doRNG
      ggplot2
    #   ggridges
    #   ggthemes
    #   outbreaks
    #   rstan
    #   pkgs.sundials
    #   (buildRPackage {
    #     name = "rethinking";
    #     src = pkgs.fetchFromGitHub {
    #       owner = "rmcelreath";
    #       repo = "rethinking";
    #       rev = "d0978c7f8b6329b94efa2014658d750ae12b1fa2";
    #       sha256 = "1qip6x3f6j9lmcmck6sjrj50a5azqfl6rfhp4fdj7ddabpb8n0z0";
    #     };
    #     propagatedBuildInputs = [ coda MASS mvtnorm loo shape rstan dagitty ];
    #   })
    #   (buildRPackage {
    #     name = "smcsamplers";
    #     src = pkgs.fetchFromGitHub {
    #       owner = "pierrejacob";
    #       repo = "smcsamplers";
    #       rev = "097192f7d5df520d9b026d442dfec493a3051374";
    #       sha256 = "00facn1ylcbai4sbcidpp991899csz2ppmmkv0khvqxfncddr0f2";
    #     };
    #     propagatedBuildInputs = [ coda MASS mvtnorm loo shape rstan dagitty ];
    # })
    ]; };

  haskellDeps = ps: with ps; [
    hmatrix-sundials
    cassava
    bytestring
    katip
    vector
    QuickCheck
    random-fu
    Cabal
    (pkgs.haskell.lib.dontCheck inline-r)
  ];

  environment.systemPackages = with pkgs; [ julia_17-bin ];

in

pkgs.stdenv.mkDerivation {
  name = "xxx";

  buildInputs = [
    pkgs.libintlOrEmpty
    R-with-my-packages
    pkgs.pandoc
    pkgs.texlive.combined.scheme-full
    (pkgs.myHaskellPackages.ghcWithPackages haskellDeps)
  ];
  shellHook = ''
    R_LIBS_USER=$(Rscript -e ".libPaths()" | cut -c 5- | Rscript -e ".libPaths()" | cut -c 5- | sed 's/[ \t]*$//' | sed 's/"//g' | sed -z 's/\n/:/g;s/:'''$/\n/' | sed 's/ //g')
    export R_LIBS_USER
    '';
}