Lyoness / mull

Mutation testing system built on top of LLVM.

Mull

See LLVM-based Mutation Testing System. Request For Comments and FOSDEM 2017: Mutation Testing, Leaving the Stone Age by Alex Denisov.

Getting started

Mull is being developed against OSX. Also it is known to work against Linux Ubuntu 16.04 and CentOS 7.

How to run Mull

Notes

Use make -f Makefile.macos help to see the common tasks used by developers.

Use MullUnitTests scheme to run the tests. Use mull-driver scheme to build Mull.

Supported mutation operators

• Math: Add,Sub,Mul,Div: replaces + with -, - with +, * with /, / with *.
• Negate Condition: inverses conditions like A -> !A or == -> !=.
• Remove Void Function Call: removes a function which doesn't have a return value.
• Replace Call: replaces function calls with Int, Float and Double return values with number 42. Alpha state, not enabled by default.
• Scalar Value: replaces 0 with 1, other numbers with 0. Alpha state, not enabled by default.
• AND <-> OR: replaces && -> || and vice versa. Alpha state, not enabled by default.

Open questions

There are the questions we want to answer with Mull eventually.

• Can mutation testing actually detect serious errors in the real-world programs?
• What are the most effective mutation operators?
• Is there any equivalence between mutation testing and different kinds of testing coverage especially MC/DC coverage?
• Can mutation testing be combined with other techniques like symbolic execution, fuzz testing, genetic programming, etc., to help in automatic test generation?

Useful Material

Papers

An Analysis and Survey of the Development of Mutation Testing

Are Mutants a Valid Substitute for Real Faults in Software Testing?

Articles

LLVM-based Mutation Testing System. Request For Comments

Mull and Klee, part 1: mutation testing analysis for Klee's Tutorial Two

Using mutation testing score to measure a reliability of a single function

Stack Exchange

Mutation testing: is it useful for assessing test quality?

Talks

FOSDEM 2017: Mutation Testing, Leaving the Stone Age by Alex Denisov

LLVM Social Berlin #6: Mull meets Rust

Get in touch

The authors and developers of Mull are Alex Denisov [e-mail] and Stanislav Pankevich [e-mail].

Feel free to contact us if you have any questions about Mull and mutation testing or any other topics related to software reliability and software in general.

Copyright

Copyright (c) 2018 Alex Denisov and Stanislav Pankevich. See LICENSE for details.