HotTest.jl - Keep your tests hot!
Using ]test (or Pkg.test()) in Julia can be annoying, because each run starts a new Julia session.
That means compilation delays, and you don't want to wait long while testing to keep feedback loops short.
Another drawback of ]test is that it always runs all tests specified in test/runtests.jl.
This means that even if the delay until the tests start isn't that long, you will have to wait until all tests were successfully completed.
HotTest.jl is an experimental package which works in conjunction with Julia's default testing pipeline in Pkg.jl. The tests are run in a sandbox module in the current session, this means that compilation delays only matter for the first run, but each run is still independent from session state. Rerunning the tests afterwards should be quick.
Also, HotTest.jl can filter out testsets by name using regular expressions, so you can choose to run only a subset of them, shortening your waiting periods further. This also works for nested testsets.
Example code
First, you need to activate a test environment so that the tests' dependencies can be loaded correctly.
HotTest.jl reexports TestEnv.activate() for this purpose:
using HotTest
# you should be cd'ed into the package's root directory that you want to test
HotTest.activate()
# print a list of all tests that HotTest.jl can find
HotTest.list()
# run all tests in `test/runtests.jl`
HotTest.test()
# run all tests in a manually specified location
HotTest.test("path_to/some_file.jl")
# only run testsets with `xy` or `xz` in their title
HotTest.test(; filter = r"x[yz]")
# if you only want to check if a substring occurs, you don't need a regex
HotTest.test(; filter = "xyz")
# specify filters for levels of nested testsets using a tuple.
# you can use "" as a shortcut to accept anything
HotTest.test(; filter = ("abc", "", r"x[yz]"))
# don't run children below nesting level 3
HotTest.test(; filter = ("abc", "", r"x[yz]"), run_children = false)Example video
Here is a screen recording of a session in which I use HotTest.jl to run some tests of GridLayoutBase.jl.
HotTest.jl.Example.mov
How it works
The code in test/runtests.jl is included after applying a transformation function to each
expression.
This function replaces all occurrences of @testset macros with if else statements
which only allow the testsets to run if their names match the filter.
In order to be able to correctly transform and match nested testsets, any occurrence of
include("some_file.jl") is resolved immediately at parse time and replaced with the code it references.
This means that HotTest.jl can only work correctly if there are no dynamic include statements.
Currently, only string literals inside include are accepted.
Also, the @testset macros cannot be spliced in by other macros, all of them must be present in the actual source code.