Inherited nested tests do not trigger refresh
Zincfox opened this issue · comments
I believe that when nested tests are discovered through inheritance (abstract A has nested N, subclass S inherits nested N) the nested tests do not respect RefreshMode.BEFORE_EACH_METHOD.
The original project where I encountered this was a kotlin-spring project with Flyway and the R2DBC adapter from #121, though I could reproduce this issue with jdbc (see below).
There it could be seen that as long as the nested tests do not modify the database, all nested tests can access the migrated and filled-with-data (via @Sql(scripts=...)) database as expected.
But once one test is added that modifies the data, all tests after it operate only the migrated version of the database - until the test execution leaves the nested class, where refreshing resumes before each test as intended.
It later turned out that the database only contains data when another test of the Nest-Hosting subclass is executed before the nested ones - if not, the nested classes receive only the migrated-but-not-filled database.
In our tests we passed references to the ApplicationContext between the nesting-levels instead of using spring-injection, which could be considered bypassing spring and as such might be an important detail in this issue.
The only workaround I could discover was making the nested classes (A.N) in the abstract host-class abstract as well and then explicitly subclass them in the subclass again (S extends A, A.N is nested in A, S.N extends A.N, S.N is nested in S), which however defeats the purpose of inheriting the nested classes.
Adding @Sql annotations to the nested classes in the abstract class appear to be executed against the unmigrated database (our schema cannot be found and its tables do not exist in the default schema either), even when @AutoConfigureEmbeddedDatabase and @FlywayTest are added as well.
Sadly I have had a lot of trouble reducing this problem out of our project-code, as such the reproducer I wound up with is still a bit long:
public class NestedTestBaseExample {
@SpringBootTest
@FlywayTest
@Sql(statements = {"INSERT INTO app.demo_table (id) VALUES (1),(2),(3);"})
@AutoConfigureEmbeddedDatabase(refresh = AutoConfigureEmbeddedDatabase.RefreshMode.BEFORE_EACH_TEST_METHOD)
static class TestContainerHost {
private final DataSource hostSource;
public TestContainerHost(@Autowired DataSource hostSource) {
this.hostSource = hostSource;
}
@Nested
class TestContainerImpl extends TestContainerBase {
public TestContainerImpl() {
super(hostSource);
}
@Test
public void OutsideTest() throws SQLException{
SubTest.assertSelectResults(
hostSource,
1,
2,
3
);
}
}
}
static abstract class TestContainerBase {
private final DataSource dataSource;
public TestContainerBase(DataSource dataSource) {
this.dataSource = dataSource;
}
@Nested
@AutoConfigureEmbeddedDatabase(refresh = AutoConfigureEmbeddedDatabase.RefreshMode.BEFORE_EACH_TEST_METHOD)
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
class SubTest {
public static void assertSelectResults(DataSource dataSource, int... expected) throws SQLException {
IntStream.Builder builder = IntStream.builder();
try (Statement statement = dataSource.getConnection().createStatement()) {
Assertions.assertTrue(statement.execute("SELECT (id) FROM app.demo_table;"), "SELECT reported no results");
ResultSet resultSet = statement.getResultSet();
SQLWarning warnings = resultSet.getWarnings();
if (warnings != null) {
Assertions.fail(warnings);
}
while (resultSet.next()) {
builder.add(resultSet.getInt(1));
}
}
int[] results = builder.build().toArray();
Assertions.assertArrayEquals(expected, results, "Got unexpected results: " + Arrays.toString(results));
}
@Test
@Order(-5)
public void selectElementsBeforeAllInsertTests() throws SQLException {
assertSelectResults(dataSource, 1, 2, 3);
}
@RepeatedTest(4)
@Order(0)
public void insertElementTest(RepetitionInfo info) throws SQLException {
int insertValue = info.getCurrentRepetition() + 3;
try (Statement insertStatement = dataSource.getConnection().createStatement()) {
Assertions.assertFalse(insertStatement.execute(
"INSERT INTO app.demo_table VALUES (" + insertValue + ");"),
"INSERT reported results instead of update-count");
Assertions.assertEquals(1, insertStatement.getUpdateCount());
}
assertSelectResults(dataSource, 1, 2, 3, insertValue);
}
@Test
@Order(5)
public void selectResetElementsAfterAllInsertTests() throws SQLException {
assertSelectResults(dataSource, 1, 2, 3);
}
}
}
}Which results in the following test-results:
Where repetition 2 retains the new element from repetition 1, repetition 3 those from rep. 2 and 1, rep. 4 of rep. 3, 2 and 1, and selectResetElementsAfterAllInsertTests operates on the data left by repetition 4.
I have created a gist where I added other files that could be relevant to this problem (build.gradle, application.properties etc.)
The behavior with "if no outside tests are executed before the nested tests" can be seen when deleting the NestedTestBaseExample.TestContainerHost.TestContainerImpl.OutsideTest() test - this results in not only all tests starting with the second iteration of insertElementTest failing, but those before that as well as the database contains no data.
There is one more 'gotcha' I would like to point out because it caused me a lot of headache while trying to reproduce this: It is important that the org.junit.jupiter.api.Test-Annotation is used because the org.junit.Test-Annotation appears to not support (inherited?) nested testing and as such will report that no tests could be found in this case.
Hi @Zincfox, thanks for the comprehensive description of the problem.
I've been digging into it for a while and realized that the root cause of this strange behavior is just the way how spring tests work. If you have several test classes (including the nested ones), you should expect that each of them is being executed in a different spring context. Note that there is a caching mechanism for reusing contexts among tests, but you can't count on it. So in your case, you have three different test classes with potentially three different contexts TestContainerHost, TestContainerImpl and SubTest. The first two classes share the same configuration parameters, so the context is reused in this case, the data source refers to the same database and everything works as expected. However, the nested test in the abstract class has no relation to the configuration parameters of the TestContainerHost class. That results in the creation of a new context and this new context contains a different data source bean referring to a different database instance.
In other words, you are passing a data source bean from the TestContainerHost context to another one with different setup and expecting the tests in SubTest to be refreshing the passed data source, but in reality the @AutoConfigureEmbeddedDatabase annotation on the SubTest class is linked to and refreshes a different data source and a different database instance. You can verify it by invoking hostSource.unwrap(EmbeddedDatabase.class).getJdbcUrl() and getting real connection details to target databases. See the attached example below.
So my recommendation is to simplify the hierarchy of test classes. Because from my point of view the @AutoConfigureEmbeddedDatabase annotation behaves correctly, just like any other spring annotations. And for instance, the behavior of Spring's @Sql annotation is the same in this case. The @Sql annotation also has no idea about the passed data source and executes all statements on the undesired data source.
@Nested
@AutoConfigureEmbeddedDatabase(refresh = AutoConfigureEmbeddedDatabase.RefreshMode.BEFORE_EACH_TEST_METHOD)
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
class SubTest {
@Autowired
private DataSource dataSource;
@Test
@Order(-5)
public void selectElementsBeforeAllInsertTests() throws SQLException {
assertSelectResults(hostSource, 1, 2, 3);
}
@RepeatedTest(4)
@Order(0)
public void insertElementTest(RepetitionInfo info) throws SQLException {
// the following two lines should print different connection string to different database instances
System.out.println(hostSource.unwrap(EmbeddedDatabase.class).getJdbcUrl());
System.out.println(dataSource.unwrap(EmbeddedDatabase.class).getJdbcUrl());
int insertValue = info.getCurrentRepetition() + 3;
try (Statement insertStatement = hostSource.getConnection().createStatement()) {
Assertions.assertFalse(insertStatement.execute(
"INSERT INTO app.demo_table VALUES (" + insertValue + ");"),
"INSERT reported results instead of update-count");
Assertions.assertEquals(1, insertStatement.getUpdateCount());
}
assertSelectResults(hostSource, 1, 2, 3, insertValue);
}
@Test
@Order(5)
public void selectResetElementsAfterAllInsertTests() throws SQLException {
assertSelectResults(hostSource, 1, 2, 3);
}
}
Thank you, that makes a lot of sense to me.
And I agree that this sounds way more like a 'me' problem and not something that needs fixing on your end, so I'll go ahead and close this issue for now.
At least in my concrete usecase (not the simplified one I posted) I think inheriting nested tests still allows for a more structured codebase, so I will probably tinker around with this a bit more and if all else fails stick with the workaround of nested implementations for abstract nested tests.
If I can find a better solution for my workaround I'll post it here so that others encountering this problem can hopefully save themselves some hassle :)