ResourceReaper too aggressive for parallel tests with singleton container
Chrisss93 opened this issue · comments
This might be more for testcontainers-java, but I'm using this library in a setup unique to the scala ecosystem so I thought I'd ask this here first. Using the scalatest framework, I'm trying to run multiple tests in the same test-class in parallel but using the same container instance. This is a common singleton pattern - which I've reproduced with a toy example:
package org.example
import com.dimafeng.testcontainers.{ForAllTestContainer, GenericContainer}
import org.scalatest.ParallelTestExecution
import org.scalatest.funsuite.AnyFunSuite
import org.slf4j.LoggerFactory
import org.testcontainers.containers.wait.strategy.Wait
import java.net.URL
import scala.io.Source
import scala.util.{Failure, Success, Try}
object ExampleTest {
val container: GenericContainer = GenericContainer("nginx:1.9.4",
exposedPorts = Seq(80),
waitStrategy = Wait.forHttp("/")
)
val log = LoggerFactory.getLogger(this.getClass)
}
class ExampleTest extends AnyFunSuite with ForAllTestContainer with ParallelTestExecution {
override val container = ExampleTest.container
test("Example") {
Thread.sleep(2000)
Try(new URL(s"http://${container.containerIpAddress}:${container.mappedPort(80)}")) match {
case Success(url) => ExampleTest.log.info(Source.fromInputStream(url.openConnection.getInputStream).mkString)
case Failure(e) => ExampleTest.log.error(e.getMessage)
}
}
}I am invoking the test from a maven project via the scalatest-maven-plugin. What I observe is that when I tell scalatest to run this in parallel, I see that the org.testcontainers.utility.ResourceReaper prematurely kills the container before the test is finished running. This behaviour does not occur when scalatest runs suites in sequence. Is this known behaviour?
Here are the logs when running this in parallel:
00:31:54.202 [ScalaTest-2] INFO 🐳 [nginx:1.9.4] - Creating container for image: nginx:1.9.4
00:31:54.248 [ScalaTest-2] INFO 🐳 [nginx:1.9.4] - Starting container with ID: 4337f6208b704b9047aa5ab7b58359c72fd359331ecde78935da07dba612bf48
00:31:54.531 [ScalaTest-2] INFO 🐳 [nginx:1.9.4] - Container nginx:1.9.4 is starting: 4337f6208b704b9047aa5ab7b58359c72fd359331ecde78935da07dba612bf48
00:31:54.548 [ScalaTest-2] INFO org.testcontainers.containers.wait.strategy.HttpWaitStrategy - /condescending_bhaskara: Waiting for 60 seconds for URL: http://localhost:53787/ (where port 53787 maps to container port 80)
00:31:54.556 [ScalaTest-2] INFO 🐳 [nginx:1.9.4] - Container nginx:1.9.4 started in PT0.37625S
00:31:54.738 [ScalaTest-2] DEBUG org.testcontainers.utility.ResourceReaper - Removed container and associated volume(s): nginx:1.9.4
00:31:56.596 [ScalaTest-3-running-ExampleTest] ERROR org.example.ExampleTest$ - Mapped port can only be obtained after the container is started
Here are the logs without scalatest parallel behaviour:
00:31:36.388 [ScalaTest-2] INFO 🐳 [nginx:1.9.4] - Creating container for image: nginx:1.9.4
00:31:36.433 [ScalaTest-2] INFO 🐳 [nginx:1.9.4] - Starting container with ID: a5b12d1233ad63b4ab415b0d3d9b7462175b15b2bd0cc17bb018beb66e94c732
00:31:36.721 [ScalaTest-2] INFO 🐳 [nginx:1.9.4] - Container nginx:1.9.4 is starting: a5b12d1233ad63b4ab415b0d3d9b7462175b15b2bd0cc17bb018beb66e94c732
00:31:36.738 [ScalaTest-2] INFO org.testcontainers.containers.wait.strategy.HttpWaitStrategy - /reverent_tereshkova: Waiting for 60 seconds for URL: http://localhost:53781/ (where port 53781 maps to container port 80)
00:31:36.746 [ScalaTest-2] INFO 🐳 [nginx:1.9.4] - Container nginx:1.9.4 started in PT0.381287S
00:31:38.789 [ScalaTest-2-running-ExampleTest] INFO org.example.ExampleTest$ - <!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
- Example
00:31:38.984 [ScalaTest-2] DEBUG org.testcontainers.utility.ResourceReaper - Removed container and associated volume(s): nginx:1.9.4
For completeness, this is the minimal pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>testcontainer-parallel-bug</artifactId>
<version>0.0.1-SNAPSHOT</version>
<properties>
<scalatest.parallel>true</scalatest.parallel>
</properties>
<dependencies>
<dependency>
<groupId>com.dimafeng</groupId>
<artifactId>testcontainers-scala-scalatest_2.13</artifactId>
<version>0.39.12</version>
</dependency>
<dependency>
<groupId>org.scalatest</groupId>
<artifactId>scalatest_2.13</artifactId>
<version>3.1.0</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.10</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.scalatest</groupId>
<artifactId>scalatest-maven-plugin</artifactId>
<version>2.0.0</version>
<configuration>
<parallel>${scalatest.parallel}</parallel>
</configuration>
<executions>
<execution>
<id>test</id>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>4.5.4</version>
<executions>
<execution>
<id>test-compile</id>
<goals>
<goal>testCompile</goal>
</goals>
<phase>test-compile</phase>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>and the examples are run as: mvn clean test and mvn clean test -Dparallel=false. Am I doing something obviously wrong here, or is there a better pattern for executing parallel tests which depend on a singleton container? Thanks!
As a simple workaround, you could disable ResourceReaper through the TESTCONTAINERS_RYUK_DISABLED environment variable.
About your approach for running multiple test cases in the single test class. Are you sure that scalatest can handle this? As far as I know in parallel mode scalatest could run multiple test classes in parallel, but not multiple test cases in the single class. At least this is how it works in the sbt.
Thanks for the quick response @LMnet! I did forget to mention I'm already running this setup with ryuk disabled. A bit further up on the log I see...
[ScalaTest-2] DEBUG org.testcontainers.DockerClientFactory - Ryuk is disabled
to confirm this. ryuk is not actually responsible for reaping testcontainers in this example. The ResourceReaper is simply creating its own docker-client to kill of the testcontainers once the GenericContainer#stop() method is called. The question I can't seem to figure out is why this stop() method is being called prematurely when scalatest runs in parallel...
Addressing your second point, I'm pretty sure for scalatest, just setting the -P flag (or equivalently the configuration.parallel property with the scalatest-maven-plugin) is what controls whether test-classes run in parallel, you are correct. But additionally you can also tell scalatest to run all tests-cases in a specific test-class in parallel by implementing a trait. The scalatest docs here state:
...
In this way, different tests of a suite that mixes in ParallelTestExecution will run in parallel.
and can be confirmed with a small snippet. When the below is run with scalatest in parallel-mode, the print-out will no longer be in alphabetical order
class Snippet extends AnyFunSuite with ParallelTestExecution {
('a' to 'd').map(_.toString).foreach(i => test(i) { println(i) })
}
when this is run with scalatest in parallel-mode, the print-out will no longer be in alphabetical order.
@Chrisss93 I can assume that the current scalatest integration is not ready for test cases parallelism. At the current moment I can't really help with this issue because I never use scalatest in such mode, and also I don't know much about scalatest and maven integration. I suggest you look at the ForAllTestContainer implementation.
Thanks @LMnet , looks like the problem is more that the scalatest's ParallelTestExecution trait to enable parallel execution of individual test-cases doesn't mix all that well with testcontainers' ForAllTestContainers trait. I've managed to accomplish this goal of re-usable containers across parallel-executed tests by defining my own trait re-implementing the most minimal degree of ForAllTestContainers logic. Surprisingly simple...probably some unaccounted for edge-cases, but it works for me.
import com.dimafeng.testcontainers.Container
import com.dimafeng.testcontainers.implicits.DockerImageNameConverters
import org.scalatest._
import org.slf4j.LoggerFactory
trait ParallelReusableContainer extends SuiteMixin with DockerImageNameConverters with BeforeAndAfterAll
with ParallelTestExecution {
this: Suite =>
def container: Container
final private val log = LoggerFactory.getLogger(this.getClass)
val testCaseData: Map[String, ConfigMap] = Map.empty
final abstract override def beforeAll(): Unit = {
super.beforeAll()
log.info("Starting container(s)")
container.start()
}
final abstract override def afterAll(): Unit = {
super.afterAll()
log.info("Closing container(s)")
container.close()
}
abstract override def run(testName: Option[String], args: Args): Status = {
if (expectedTestCount(args.filter) == 0) {
new CompositeStatus(Set.empty)
} else {
testName.flatMap(testCaseData.get) match {
case Some(c) => super.run(testName, args.copy(configMap = c))
case None => super.run(testName, args)
}
}
}
}