Giters

tarides / opam-monorepo

Assemble dune workspaces to build your project and its dependencies as a whole

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

opam-monorepo for Tezos

NathanReb opened this issue · comments

commented

We will use this issue to track all the work needed to build the tezos repo using opam-monorepo.

Building Tezos with opam-monorepo

The goal here is to make sure that it is possible to build tezos using a regular opam-monorepo workflow, using only upstream opam-repository (or for now using Tezos' repository) and opam-overlays. No local modifications should be required and neither should any opam pins.

The preliminary steps are, similarly to any opam-monorepo project:

  1. Clone the Tezos repository
  2. cd into the repo
  3. Run opam switch create ./ --empty to set up an empty local switch
  4. Setup the appropriate repos:
$ (. scripts/version.sh && opam repository add tezos git+$opam_repository)
$ opam repository add dune-universe git+https://github.com/dune-universe/opam-overlays

From there or any already properly setup repo and switch, you should be able to:

  1. Run opam monorepo lock --recurse --lockfile tezos.opam.locked and successfully get a lockfile
  2. Run opam monorepo pull to successfully setup the duniverse/
  3. Run opam install --ignore-pin-depends --deps-only ./tezos.opam.locked to opam install the rigth OCaml and dune versions in our local switch
  4. Run dune build successfully
  5. Run dune runtest successfully

UX improvement should follow but reaching this point is our first milestone.

What is left to do

As of 23/03/2022, here is the list of remaining items:

  • Relax constraint in Tezos to allow selection of dune ports. There is a PR for this here

  • Fix ometrics so that it can be used with opam-monorepo. I created an issue upstream

    • A PR has been opened on merlin. The plan is to cut a release of merlin soon to remove ometrics' vendoring of merlin.

  • Fix dune+findlib in duniverse bug. Vendored findlib does not play nice with dune at the moment. It cannot locate the standard library due to how the environment is setup by dune. That causes jsoo to crash in Tezos. There's a working fix in dune, once it's merged and released we should be good: ocaml/dune#4197

  • Properly port to dune missing packages. The following packages have existing ports but they are hosted on a repo that was maintained by Craig when he originally attempted to build Tezos with monorepo. We need to clean them up and release them to opam-overlays:

    • zarith.1.12
    • ctypes.0.18.0 or higher. Upstream's port to dune is almost ready, probably worth unblocking this as well.
    • ocp-ocamlres.0.4.0 or higher.
    • pyml. There are no version constraints on the pyml dependency in the repo. Craig's port is based on 20210226 and gets picked atm. Latest released version uses dune but also stdcompat which might explain why it's not picked. A PR exists on opam-overlays and should be merged soon.

    There are other packages that need a dune port as well:

    • hacl-star.0.4.5 and hacl-star-raw.0.4.5. We already have a working port but it needs to be refined before we can release it to opam-overlays.

  • Release opam-monorepo.0.3.0. Tezos needs the opam-provided feature to allow opam installing tezos-rust-libs. Some other usability features and improvement that will ship in that version, in particular the improved lockfile compat with opam.

  • Repackage bls12-381-legacy.0.4.4 and bls12-381-gen.0.4.4. At the moment, Tezos depends on these and more recent versions that are maintained in the same repository. That prevents opam-monorepo from pulling a single version of the tarball that contains all packages. This dependency should be dropped soon by Tezos so releasing the repackaged repo we already have might not be necessary.

We will keep this list up to date on a regular basis and add any new item that ermeges as we progress.

commented

The biggest blocker atm is the JSOO bug in a duniverse as it's the only thing that prevents us from building the whole repo.

Most of the package above have draft dune ports that we currently use to run the builds so it's a matter of officially releasing them.

The ometrics issue needs to be properly addressed but it's not preventing us from progressing as it can be worked around by removing the duplicate merlin repo from the duniverse/ folder after pull.

commented

I just unticked the hacl-star ports as it seems that the ports trigger build failure on tezos in their current state. We're investigating this to fix the port!

commented

Updated on JSOO: the bug was unrelated to jsoo itself but was in fact a bug with vendored findlib and ocamlfind.

This was a more recent instance of an older bug in dune. There's a fix PR here: ocaml/dune#4197.

Getting this merged and release should solve this!

commented

We can close this, the PR had been opened on the Tezos repository.

commented

Could you link the relevant PR?