dune install wants opam or --prefix

[Jerry James]

Expected Behavior

With dune 2.x, we could run "dune install --destdir=" when building Fedora packages with dune, and that would populate our buildroot as expected.

Actual Behavior

With dune 3.0.2, the same dune invocation results in this error:

Error: Program opam not found in the tree or in PATH

That is not helpful when we are trying to build the dependencies of opam. I have found that if I add --prefix=/usr, then dune stops asking for opam ... but it installs the library files into the wrong directory. We configured dune with --libdir, but that setting is ignored if the prefix is given. So now I have to pass both --prefix and --libdir to dune install, even though dune should already know the correct libdir.

Reproduction

1. Build any project with dune
2. Ensure that opam is not in PATH
3. Make an install directory, say /tmp/dune-root
4. Run "dune install --destdir=/tmp/dune-root"

Specifications

• Version of dune (output of dune --version): n/a <-- That is the actual output of dune --version. It is really 3.0.2.
• Version of ocaml (output of ocamlc --version): 4.13.1
• Operating system (distribution and version): Fedora Rawhide

Additional information

• Link to gist with verbose output (run dune with the --verbose flag):
  Running dune --verbose just prints help text. Is it supposed to show configuration values? Running dune build --verbose on the other hand prints this: https://gist.github.com/jamesjer/0c2a5b82c7af23fa47e65587f6a67f6f

[rwmjones]

I'm not even sure why dune would run opam at all.

[François Bobot]

even though dune should already know the correct libdir.

Do you mean because it is given during the compilation of dune with ocaml configure --libdir <path> ?

I think the main problem is this configuration command doesn't allow to give a default --prefix so opam is used to get it, or we don't allow to give a default for all the path (e.g share). However if --prefixis set and not libdir, the configured libdir is not used. I think it is normal. When you change the prefix everything is changed.

Just to check before writing a fix do you prefer one option during the configuration of dune for each of the root directories? (e.g also one for libexec_root)? If they are all set prefix would not be needed .

(destdir in this code snippet is applied with prefix)

dune/src/dune_engine/install.ml

Lines 142 to 168 in 065466c

     (* FIXME: we should handle all directories uniformly, instead of 
        special-casing etcdir, mandir, and docdir as of today [which was done for 
        convenience of backporting] *) 
     let make ~package ~destdir ?(libdir = Path.relative destdir "lib") 
         ?(mandir = Path.relative destdir "man") 
         ?(docdir = Path.relative destdir "doc") 
         ?(etcdir = Path.relative destdir "etc") () = 
       let package = Package.Name.to_string package in 
       let lib_root = libdir in 
       let libexec_root = libdir in 
       let share_root = Path.relative destdir "share" in 
       let etc_root = etcdir in 
       let doc_root = docdir in 
       { lib_root 
       ; libexec_root 
       ; share_root 
       ; bin = Path.relative destdir "bin" 
       ; sbin = Path.relative destdir "sbin" 
       ; man = mandir 
       ; toplevel = Path.relative libdir "toplevel" 
       ; stublibs = Path.relative libdir "stublibs" 
       ; lib = Path.relative lib_root package 
       ; libexec = Path.relative libexec_root package 
       ; share = Path.relative share_root package 
       ; etc = Path.relative etc_root package 
       ; doc = Path.relative doc_root package 
       } 

CC @ejgallego

[Jerry James]

Do you mean because it is given during the compilation of dune with ocaml configure --libdir <path> ?

Yes, that is what I mean.

I think the main problem is this configuration command doesn't allow to give a default --prefix so opam is used to get it, or we don't allow to give a default for all the path (e.g share). However if --prefixis set and not libdir, the configured libdir is not used. I think it is normal. When you change the prefix everything is changed.

Sure, that makes sense. In that case, we would like to set the default prefix along with the libdir.

Just to check before writing a fix do you prefer one option during the configuration of dune for each of the root directories? (e.g also one for libexec_root)? If they are all set prefix would not be needed .

Actually, the other defaults are all fine for us. We just need to special case libdir. We can work with setting a default prefix, or with setting all of the directories individually. Either is fine. Thank you for looking at this issue.

[Kate]

I’m not sure why the opam binary is used to get the prefix either. Why is the OPAM_SWITCH_PREFIX environment variable not used instead?

[François Bobot]

Is there cases when one way will work and not the other? If OPAM_SWITCH_PREFIX is not present, opam binary could work. Is there symmetric cases?

[Guillaume Melquiond]

If OPAM_SWITCH_PREFIX is not present, opam binary could work.

But just because opam happens to be installed on the system does not mean that it should be used to decide on an installation location. If OPAM_SWITCH_PREFIX is not defined, calling opam looks like a bug to me.

[François Bobot]

Ok OPAM_SWITH_PREFIX must be used (in fact it is already the case during build).

So the priority:

• Use the command line options
• Use information given during configuration of dune
• Otherwise use OPAM_SWITCH_PREFIX to get the information.
• Otherwise fails with an error

During build ocamlc -config is used for the configuration in particular OCAMLPATH, but I prefer not to use it for finding the prefix.

Still I don't understand why in mirage/ocaml-tar#85 and #5488 , opam var prefix fails inside opam execution but we will look at the problem in #5488.

[François Bobot]

@jamesjer at the end, when everything is added and fixed in dune, it seems needed to give the full list of the directory to both the ./configure of dune and to dune install. Correct me if I'm wrong but:

• The dune package must be usable by the users to install libraries in /usr/local, so we need at least ./configure --libdir /usr/local/lib/ocaml --libdir /usr/lib/ocaml --mandir /usr/local/share/man --docdir /usr/local/share/doc ... . The option --libdir appears twice in order to indicate the installation directory and another directory for the lookup.
• Other packages compiled with dune must be installed in /usr, so we need dune install --destdir=.. --libdir /usr/lib/ocaml --mandir /usr/share/man --docdir /usr/share/doc ...

[Jerry James]

My personal preference is to use the values passed to configure as defaults. That is, if I run ./configure --prefix=/usr, then dune defaults to installing into /usr; i.e., plain dune install can be invoked and do something useful. A user can still install elsewhere by passing --prefix, --libdir, etc. on the command line.

[Blaine Bublitz]

fwiw, an esy sandbox also doesn't have the opam command and any dune install in a package fails on dune 3.

[François Bobot]

The value given to configure will be used as default. I just though that you preferred that user can do simply sudo dune install and have the files installed in /usr/local and not in /usr/. But the choice of defaults is yours and will be used.

[François Bobot]

@phated thank you for the report, #5516 should also fix esy. But where does esy install files and how does it customize the directories?

[Eduardo Rafael]

@bobot esy installs it at $cur__install on the environment, but normally we just use esy-installer something.install which will read the install file and put everything in the right place.

[Yawar Amin]

Shouldn't this be labelled bug? Dune should definitely not depend on opam, right?

[Rudi Grinberg]

Indeed. It's properly labelled now.