fact check

Midje-like testing for Julia.

Why?

Base.Test seemed to be the only option in the way of Julia testing, and I wasn't a huge fan. Midje is a simple and powerful testing framework written for Clojure, and so I sought to (at least partially) recreate it. This is a work in progress.

Installation

julia> Pkg.add("FactCheck")

Usage

using FactCheck

The top-level function facts describes the scope of your tests and does the setup required by the test runner. It can be called with or without a description:

facts("With a description") do
    # ...
end

facts() do
    # ...
end

Inside of the function passed to facts, a fact can be asserted using the @fact macro, or @fact_throws if you're asserting a thrown exception.

facts("Simple facts") do

    # expression => assertion
    @fact 1 => 1

    @fact_throws error()

end

Related facts can also be grouped inside of a context:

facts("Simple facts") do

    context("numbers are themselves") do
        @fact 1 => 1
        @fact 2 => 2
        @fact 3 => 3
    end

end

The symbol => is used as an assertion more general than ==. Each fact will be transformed into a test, the type of which depends on the value to the right of the =>. (We'll call that value the assertion.)

An assertion can take two forms:

# If the assertion is a function, it will be called on the expression
# to determine whether or not the fact holds.
@fact 2 => iseven

# Otherwise, the fact holds if the expression is `==` to the assertion.
@fact [1,2,3] => [1,2,3]

As the function-assertion form is rather convenient and reads nicely, a number of helper assertion functions are provided.

facts("Using helper assertions") do

    context("some helpers operate on values") do
        @fact false => anything
        @fact 1 => not(2)
    end

    context("... or on functions") do
        @fact 2 => not(isodd)
    end

end

These can be found at the bottom of src/FactCheck.jl.

Workflow

First, make sure your tests are inside of a module.

module TestFactCheck

# assertions live here

end # module

Then you can simply reload your test file repeatedly inside of a Julia REPL. Because the tests are in a module, you won't run into constant-redefinition errors if you create constants (like types or immutables) inside of your tests. (Julia allows you to repeatedly replace a module inside a single process, but you cannot replace constants.)

julia> reload("test_factcheck")
# Output summary...

julia> reload("test_factcheck")
Warning: replacing module TestFactCheck
# Output summary...

This workflow has the advantage of not requiring an extra invocation of julia on each test run, which would add a few seconds to your testing time.

A convenience macro named @runtest is also provided. The @runtest macro takes a package name and any number of valid test files. It expects the test files to be in the /test directory and to be appended by .jl. It simply includes the specified files, which allows multiple calls within a single Julia session.

julia> @runtest FactCheck test_factcheck

FactCheck core functions

9 facts verified.


FactCheck assertion helper functions

24 facts verified.

The macro also works on tests outside the FactCheck framework.

julia> @runtest Stats means variability

# these tests pass silently

Configuration

FactCheck has a configuration dictionary at FactCheck.config. Currently the only option is :compact, which can be true or false to turn on or off more compact output. The default is false, so to enable compact output simply put FactCheck.config[:compact] = true before your tests.

Travis

If you want to use FactCheck tests on Travis, you can emit the overall result of the tests like this:

modlue TestFactCheck

    ### your tests...

    exitstatus()

end # module

This will return 0 if everything went well, or the number of errors + failures otherwise. In your .travis.yml add a line like the following:

- julia ~/.julia/YourPackage/test/yourtests.jl

You can then include the Travis built status indicator as usual as featured on the top of this page.

Contributing

I'm incredibly open to contributions. The code base is quite small and (I think) well documented. I'm also happy to explain any decisions I've made, with the understanding that they may have been uninformed.

alabamacajun / FactCheck.jl