Dune fails to handle symlinks under windows/cygwin

[Yoann Padioleau]

I'm trying to compile semgrep (an OCaml project) under windows. I've followed
the "ocaml for windows" instructions in https://fdopen.github.io/opam-repository-mingw/installation/
and managed opam to install all the necessary packages and ocamlc to compile most files,
but I ran into issues with a few files that are symbolic links.

I have this dune rule in one my dune file:

(rule
 (targets semgrep_output_v1_j.ml semgrep_output_v1_j.mli)
 (deps    semgrep_output_v1.atd)
 (action  (run atdgen -j -j-std -j-defaults %{deps})))

where semgrep_output_v1.atd is a symlink to a file in another directory
but dune errors with

Error: no rule found for OSS/src/core/semgrep_output_v1.atd"

Note that I am able to run cat on this file, so the symlink works, but
dune does not seem to find it.

Expected Behavior

dune build should work

Actual Behavior

Error: no rule found for OSS/src/core/semgrep_output_v1.atd"

Reproduction

• PR with a reproducing test:

Specifications

• Version of dune (output of dune --version):
• Version of ocaml (output of ocamlc --version)
• Operating system (distribution and version):

Additional information

• Link to gist with verbose output (run dune with the --verbose flag):

[Rudi Grinberg]

@nojb could you confirm this issue?

[Yoann Padioleau]

Note that it seems to work from CI. I'm also using the setup-ocaml@v2 Github Action, and there is a windows-latest option,
and in that case dune is working fine. But I don't know which magic setup-ocaml@v2 is using to make dune handle the symlinks correctly. Or maybe it's checking out the code in a special way; I'm using the git from cygwin.

[Nicolás Ojeda Bär]

Note that I am able to run cat on this file, so the symlink works, but
dune does not seem to find it.

Cygwin Symlinks are complicated: Cygwin has several notions of symlinks; they can be implemented solely in the Cygwin layer or they can be native symlinks. I suspect you will have better luck with native symlinks. See https://cygwin.com/cygwin-ug-net/using.html#pathnames-symlinks. Can you try with native symlinks and see if the problem goes away? (Instructions to switch are in the link, it invoves setting the CYGWIN environment variable.)

[Yoann Padioleau]

So if I use this environment variable and rerun git clone, it will create native symlinks that dune will recognize?

[Nicolás Ojeda Bär]

So if I use this environment variable and rerun git clone, it will create native symlinks that dune will recognize?

Perhaps :) You should do the experiment and let us know! Thanks.

[Yoann Padioleau]

Ok I've made progress; using the CYGWIN=winsymlinks.native helped. I had to reclone so that git clone also create the right symlinks. I still have a problem though when the symlinks are broken at first. I'm using submodules and some of those symlinks are crossing submodules, but when I checkout first the main repo, the symlinks are originally dead. When I create the submodules then those symlinks should become alive, but windows does not seem to agree; if I use cat in powershell I get some cat: the file cannot be accessed by the system. If I delete those symlinks and recreate them with git reset --hard then it works ...

[Yoann Padioleau]

Closing seems the issue seems to be how windows handle broken symlinks. Looks like they are not "late bounded".

[Yoann Padioleau]

Not sure why it works in CI under setup-ocaml@v2 though.