moby-ryuk containers leak with SBT
alex-vv opened this issue · comments
I use version 1.6.0 with testcontainers-scala 0.14 to run scala integration tests with sbt. When running tests in the interactive mode I see that moby-ryuk container is not stopped and it is created again for each run, so after few runs docker ps shows the following:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
832de2504d77 bsideup/moby-ryuk:0.2.2 "/app" 31 seconds ago Up 52 seconds 0.0.0.0:32962->8080/tcp testcontainers-ryuk-fb8fc138-b54f-46da-ae76-f72f77ff6bc5
66330a95d32c bsideup/moby-ryuk:0.2.2 "/app" 51 seconds ago Up About a minute 0.0.0.0:32959->8080/tcp testcontainers-ryuk-1abe1265-f87b-4ca8-a426-fdc005da58d0
9ce16c780c80 bsideup/moby-ryuk:0.2.2 "/app" About a minute ago Up About a minute 0.0.0.0:32956->8080/tcp testcontainers-ryuk-6f6c27a5-cd03-4c1c-b6b0-e8f3633c2bfe
e6af83fa93f6 bsideup/moby-ryuk:0.2.2 "/app" About a minute ago Up 2 minutes 0.0.0.0:32953->8080/tcp testcontainers-ryuk-a6b83d36-4b9b-4c93-bfd3-e37d67442092
Once I exit sbt these containers are stopped. I think this problem is not scala or sbt-specific but may appear in any case when you run a test suite repeatedly on the same JVM.
Tried version 1.5.1 and it doesn't have this problem. I guess the issue is that ResourceReaper is started in DockerClientFactory but I could not find that it is stopped anywhere.
@alex-vv this container will not exit until you have an open TCP connection to it, but rest assured it will once you close it. It seems that SBT keeps some JVMs running, maybe daemon thing or something, this is why it happens.
ryuk containers are very lightweight, implemented in Golang and will not consume too many resources on your machine, you can safely ignore them since there is no leak (as you said they disappear once you exit sbt)
FYI 1.5.1 doesn't have it because we introduced Ryuk in 1.6.0 :)
Yes, in the interactive mode sbt keeps JVM running and that is expected behavior. Do I understand correctly that TCP connection to ryuk container is managed internally by DockerClientFactory and there is no way for the library user to manually close it without stopping JVM?
It is good that ryuk is lightweight but if I re-run tests continuously it still doesn't feel right to have a lot of them hanging around. Maybe if you don't want to automatically stop it, that's ok, but is it possible to provide some API to do that manually if needed?
@alex-vv I'm quite sure you will have as many Ryuks as JVMs in SBTs, so it's not "a lot" but 5-8 or how many JVMs SBT keeps running?
The problem is that if you kill them manually and then some of the tests will run on that daemonized JVM your containers will not get cleaned.
@alex-vv I'm quite sure you will have as many Ryuks as JVMs in SBTs, so it's not "a lot" but 5-8 or how many JVMs SBT keeps running?
no, in my case a new ryuk gets created for each new test run, and they are all in one JVM. I can try to do some minimal reproducible case if needed.
@bsideup you can check https://github.com/alex-vv/testcontainers-sbt
@alex-vv thanks, I'll check it soon 👍
@alex-vv we use test containers with sbt and Scala and saw similar problems. What we did was essentially ‘fork in Test := true’ which ensured that the test itself ran in a separate JVM and when exited, killed off the containers. At this point, sbt in interactive mode is still running.
I believe I have the same issue with Gradle and Testcontainers 1.9.rc-1. Ryuk containers just stay lingering after test completes. Or maybe only after test is canceled. Anyway I have now 48 Ryuk containers :) In status "Exited (0)"
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this is a mistake, please reply to this comment to keep it open. If there isn't one already, a PR to fix or at least reproduce the problem in a test case will always help us get back on track to tackle this.
+1 for fixing this, I don't want to fork tests for the sole purpose of killing the containers after the tests. Perhaps there could be a method for killing them manually?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this is a mistake, please reply to this comment to keep it open. If there isn't one already, a PR to fix or at least reproduce the problem in a test case will always help us get back on track to tackle this.
I updated my test case here https://github.com/alex-vv/testcontainers-sbt with the latest versions of sbt and testcontainers-scala and this issue still persists. Currently I run all integration tests in a forked jvm so it doesn't affect me anymore, but posting this since the original issue is not fixed yet.
I believe we're running into this issue as well.
We're using testcontainers in our gradle process to set up a PostgreSQL container that JOOQ's codegen can run against. We're able to shut down the PostgreSQL container, but the ryuk container continues running, presumably because the gradle daemon's JVM is still running.
If we start another PostgreSQL container in another gradle build, another ryuk container shows up in docker ps. This is true even with only one gradle daemon JVM running. I'm guessing each ryuk was started by a different classloader.
I tried doing this 20 times in a row, and the number of ryuks only went up to 6. So it isn't a disaster, but it would still be nice if there was a way to shut these things down when the build finishes.
Would it be possible to create a "shut everything down" method in testcontainers that would stop all of the containers including ryuk?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this is a mistake, please reply to this comment to keep it open. If there isn't one already, a PR to fix or at least reproduce the problem in a test case will always help us get back on track to tackle this.
This issue has been automatically closed due to inactivity. We apologise if this is still an active problem for you, and would ask you to re-open the issue if this is the case.
Three years later we are still facing this issue. Shouldn't we keep it open?
@ghostbuster91 do you have a reproducer of the leak? (with the latest version of Testcontainers, 1.17.2)
Hi,
I am using testcontainers-scala 0.40.5 that is based on testcontainers 1.16.2.
I know hat it can be solved by running tests in the fork mode, but I would rather avoid doing that if I could.
Has anything related to how ryuk containers operate changed between 1.17 and 1.16?