cafejojo / test-suite-novelty

Tool for determining the novelty of a test suite with respect to a baseline test suite

Test Suite Novelty Analyzer

Tool for determining the novelty of a test suite with respect to a baseline test suite. It compares two JUnit test suites (a "new" one and an "old" one) test case by test case. For each, it analyzes the set of items covered by each test in the new test suite against each old test. If none of the sets of items covered is a subset of a certain old test case, that test case is considered "new". The output of this tool gives the number of novel test cases.

Requirements and Installation

This system requires JRE 8 and has been tested on Windows and Unix systems.

Usage

This tool assumes the following directory structure of Maven project to be analyzed:

src/
- main/
- test.old/
- test.new/
pom.xml

The pom.xml needs to have at least the following configurations:

<build>
    <!-- Only include this section if you need to have other directories on the classpath during build time. -->
    <resources>
        <resource>
            <directory>{path to your any other directory you want to have on the build classpath}</directory>
        </resource>
    </resources>

    <plugins>
        <!-- Test runner plugin -->
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>3.0.0-M1</version>
            <configuration>
                <argLine>${surefireArgLine}</argLine>
                <reuseForks>true</reuseForks>
            </configuration>
        </plugin>

        <!-- Coverage report plugin -->
        <plugin>
            <groupId>org.jacoco</groupId>
            <artifactId>jacoco-maven-plugin</artifactId>
            <version>0.8.2</version>
            <executions>
                <execution>
                    <id>pre-unit-test</id>
                    <goals>
                        <goal>prepare-agent</goal>
                    </goals>
                    <configuration>
                        <destFile>${project.build.directory}/coverage-reports/jacoco-ut.exec</destFile>
                        <propertyName>surefireArgLine</propertyName>
                    </configuration>
                </execution>
                <execution>
                    <id>post-unit-test</id>
                    <phase>test</phase>
                    <goals>
                        <goal>report</goal>
                    </goals>
                    <configuration>
                        <dataFile>${project.build.directory}/coverage-reports/jacoco-ut.exec</dataFile>
                        <outputDirectory>${project.reporting.outputDirectory}/jacoco-ut</outputDirectory>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

Run the tool with the following command:

./gradlew run --args='path/to/your/project'

Changelog

Please see releases for more information on what has changed recently.

Testing

$ ./gradlew check

Documentation

$ ./gradlew dokka

Contributing

Please see CONTRIBUTING for details.

Security

If you discover any security-related issues, please email [e-mail] instead of using the issue tracker.

License

The MIT License (MIT). Please see the license file for more information.