Junit5 nested test classes run twice

[Mason Chen]

Tested with JUnit5 5.9.1 and 5.10.2.

Steps to reproduce

public class MasonTest {

    @Nested
    class MasonTest1 {
        @Test
        void test1() {
            System.out.println("test1");
        }
    }

    @Nested
    class MasonTest2 {
        @Test
        void test1() {
            System.out.println("test2");
        }
    }
}

And running mvn -Dtest=MasonTest\* verify -pl flink-connector-kafka results in the logs:

[INFO] Running org.apache.flink.streaming.connectors.kafka.MasonTest
[INFO] Running org.apache.flink.streaming.connectors.kafka.MasonTest$MasonTest2
[INFO] Running org.apache.flink.streaming.connectors.kafka.MasonTest
[INFO] Running org.apache.flink.streaming.connectors.kafka.MasonTest$MasonTest2
test2
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.048 s - in org.apache.flink.streaming.connectors.kafka.MasonTest$MasonTest2
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.068 s - in org.apache.flink.streaming.connectors.kafka.MasonTest
test2
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.051 s - in org.apache.flink.streaming.connectors.kafka.MasonTest$MasonTest2
[INFO] Running org.apache.flink.streaming.connectors.kafka.MasonTest$MasonTest1
test1
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 s - in org.apache.flink.streaming.connectors.kafka.MasonTest$MasonTest1
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.085 s - in org.apache.flink.streaming.connectors.kafka.MasonTest
[INFO] Running org.apache.flink.streaming.connectors.kafka.MasonTest
[INFO] Running org.apache.flink.streaming.connectors.kafka.MasonTest$MasonTest1
test1
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.041 s - in org.apache.flink.streaming.connectors.kafka.MasonTest$MasonTest1
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.063 s - in org.apache.flink.streaming.connectors.kafka.MasonTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0

Context

• Used versions (Jupiter/Vintage/Platform): Jupiter
• Build Tool/IDE: maven

Deliverables

• ...

[Marc Philipp]

@mas-chen Please provide a complete reproducer, i.e. including the project's POM etc.

[Mason Chen]

Sure here is my reproducer test. https://github.com/mas-chen/flink-connector-kafka/blob/b93fc74029a7c88c61ed04dc820fc9f59d554448/flink-connector-kafka/src/test/java/org/apache/flink/streaming/connectors/kafka/MasonTest.java

Surefire is configured as part of an external project (see parent pom in the root pom)

[Marc Philipp]

Thanks!

I debugged the execution. In this case, Maven Surefire sends three separate discovery/execution requests (order is unclear to me, might be dependent on file system):

1. ClassSelector [className = 'org.apache.flink.streaming.connectors.kafka.MasonTest$MasonTest1']
2. ClassSelector [className = 'org.apache.flink.streaming.connectors.kafka.MasonTest']
3. ClassSelector [className = 'org.apache.flink.streaming.connectors.kafka.MasonTest$MasonTest2']

(1) will cause MasonTest1 to be executed, (3) will cause MasonTest2 to be executed, while (2) will cause both to be executed. That explains the double execution. IIRC Surefire scans the test classes dir (target/test-classes) for classes matching MasonTest* (supplied via -Dtest). Thus it finds MasonTest.class but also MasonTest$MasonTest1.class and MasonTest$MasonTest2.class. Interestingly, this does not happen when running all tests (without passing -Dtest). In any case, I think Surefire should filter out nested classes when the enclosing class passes the filter to avoid this. Please raise an issue with the Maven Surefire project.

[Mason Chen]

@marcphilipp thanks for the pointers, this helps a lot! I'll try an exclude filter on the nested classes