[Proposal] Add a new `opam test` subcommand and change the behaviour of the `run-test` field
kit-ty-kate opened this issue · comments
The issues with the current design
Testing packages in opam isn't really a separate step of action and this can cause issues for some packages.
For instance, for some packages (key packages mainly), their tests do not get defined in opam-repository because they require packages that require the package being tested (cyclic dependency). e.g. base
, odoc
, …
Some packages also do not define their tests for other reasons but opam cannot distinguish between "has test" and "does not have test" and thus users using opam install --with-test <pkg>
won't know if <pkg>
has really been tested.
The proposal
To help with those issues my proposal would be to add a new test
subcommand such as opam test <pkg>
would:
- Install the dependencies and test dependencies
- If
<pkg>
is already installed and{with-test}
is used inside thebuild
step:- then rebuild
<pkg>
without reinstalling it - else: do not rebuild it
- then rebuild
- If
<pkg>
is not installed, install it withwith-test = true
- Wait for everything to be installed (including
post
dependencies) - Run the
run-test
phase (wasn't used in any previous step) - If no tests has been ran, opam would print some kind of a note to notify the user of that.
The new run-test
step is a change of behaviour from previous versions of opam where the run-test
phase was ran after the build
phase and before the install
phase. Now with this proposal, this phase is never ran through opam install
but only through the new opam test
command.
This feels very like the opam build
experiment - moving opam into an area it simply doesn't belong in. A little like the exception
type in OCaml before extensible-variants, --with-test
and --with-doc
are both poor person's package variables (which are on the slate for "opam.2.soon"!). A few thoughts:
- The second "if pkg is already installed" stage can, I think, already be achieved by splitting the package into
foo
andfoo-tests
(potentially with an alteration to a package's test suite to test installed binary artefacts, rather than build ones - although with Dune I think would be transparent). Either way, it feels like packages should be bending to what opam offers, rather than the other way around! - I'm not sure you can come up with a mechanical definition of what tests being run would entail - opam doesn't know if a build has done anything meaningful either. For example, I notice batteries defines test dependencies but doesn't actually have
[make "test"] {with-test}
in its build recipe. However, it's not necessarily a bug - it has a configure command and there's absolutely no way opam could know what the[make "all"]
it does have would also run the testsuite if the test dependencies are installed. - It could be worth displaying notes/warnings when
--with-test
is specified but packages have neither test dependencies nor any parts of a build recipe guarded withwith-test
This is a good candidate for a plugin, using the x-*
fields.
related to #4099 ?