[1.6.4/Scala] Testkit starts up Akka cluster for StandaloneLagomClientFactory when not requested

Question

[1.6.4/Scala] Testkit starts up Akka cluster for StandaloneLagomClientFactory when not requested

solarmosaic-kflorence opened this issue 3 years ago · comments

Kyle Florence commented 3 years ago

Lagom Version (1.2.x / 1.3.x / etc)

1.6.4

API (Scala / Java / Neither / Both)

Scala

Operating System (Ubuntu 15.10 / MacOS 10.10 / Windows 10)

Mac OS 10.15.7

JDK (Oracle 1.8.0_112, OpenJDK 1.8.x, Azul Zing)

java version "1.8.0_172"
Java(TM) SE Runtime Environment (build 1.8.0_172-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode)

Library Dependencies

"com.lightbend.lagom" %% "lagom-scaladsl-testkit" % "1.6.4"

Expected Behavior

When creating a client with StandaloneLagomClientFactory, unless explicitly requested an Akka cluster is not bootstrapped.

Actual Behavior

When the client starts up, an Akka cluster is bootstrapped:

[INFO ][2021-03-31 17:32:54,771][akka.remote.artery.tcp.ArteryTcpTransport][application-akka.actor.default-dispatcher-5][akkaAddress=akka://application@192.168.1.15:25520, sourceThread=pool-7-thread-4-ScalaTest-running-ExampleValidLagomOpenApiCompatibilitySpec, akkaSource=ArteryTcpTransport(akka://application), sourceActorSystem=application, akkaTimestamp=22:32:54.771UTC] Remoting started with transport [Artery tcp]; listening on address [akka://application@192.168.1.15:25520] with UID [-6744528107701742272] [---]
[INFO ][2021-03-31 17:32:54,796][akka.cluster.Cluster][application-akka.actor.default-dispatcher-5][akkaAddress=akka://application@192.168.1.15:25520, sourceThread=pool-7-thread-4-ScalaTest-running-ExampleValidLagomOpenApiCompatibilitySpec, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=22:32:54.796UTC] Cluster Node [akka://application@192.168.1.15:25520] - Starting up, Akka version [2.6.8] ... [---]
[INFO ][2021-03-31 17:32:54,903][akka.cluster.Cluster][application-akka.actor.default-dispatcher-5][akkaAddress=akka://application@192.168.1.15:25520, sourceThread=pool-7-thread-4-ScalaTest-running-ExampleValidLagomOpenApiCompatibilitySpec, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=22:32:54.903UTC] Cluster Node [akka://application@192.168.1.15:25520] - Registered cluster JMX MBean [akka:type=Cluster] [---]
[INFO ][2021-03-31 17:32:54,903][akka.cluster.Cluster][application-akka.actor.default-dispatcher-5][akkaAddress=akka://application@192.168.1.15:25520, sourceThread=pool-7-thread-4-ScalaTest-running-ExampleValidLagomOpenApiCompatibilitySpec, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=22:32:54.903UTC] Cluster Node [akka://application@192.168.1.15:25520] - Started up successfully [---]
[INFO ][2021-03-31 17:32:54,931][akka.cluster.Cluster][application-akka.actor.default-dispatcher-5][akkaAddress=akka://application@192.168.1.15:25520, sourceThread=application-akka.actor.internal-dispatcher-3, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=22:32:54.930UTC] Cluster Node [akka://application@192.168.1.15:25520] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing [---]
[INFO ][2021-03-31 17:32:54,931][akka.cluster.Cluster][application-akka.actor.default-dispatcher-5][akkaAddress=akka://application@192.168.1.15:25520, sourceThread=application-akka.actor.internal-dispatcher-3, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=22:32:54.931UTC] Cluster Node [akka://application@192.168.1.15:25520] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining [---]

Reproducible Test Case

import com.lightbend.lagom.scaladsl.client.{StandaloneLagomClientFactory, StaticServiceLocatorComponents}
import org.scalatest.matchers.should.Matchers
import org.scalatest.wordspec.AnyWordSpec
import org.scalatest.BeforeAndAfterAll
import play.api.libs.ws.ahc.AhcWSComponents

import java.net.URI

class Lagom3211Tests extends AnyWordSpec with Matchers with BeforeAndAfterAll {
  private val clientFactory = new StandaloneLagomClientFactory("client")
    with AhcWSComponents
    with StaticServiceLocatorComponents {
    override def staticServiceUri: URI = new URI("http://lagomframework.com")
  }

  override protected def afterAll(): Unit = clientFactory.stop()

  "client" should {
    "use 'local' provider" in {
      val provider = clientFactory.actorSystem.settings.config.getString("akka.actor.provider")
      provider shouldBe "local"
    }
  }
}

Kyle Florence · Answer 1 · Thu Apr 01 2021 06:45:09 GMT+0800 (China Standard Time)

cc @octonato since you worked on the original PR to fix this, you might have more context.

Renato Cavalcanti · Answer 2 · Fri Apr 02 2021 23:02:47 GMT+0800 (China Standard Time)

@solarmosaic-kflorence, this is not what I observed.

Please check this minimal hello-world I published here: https://github.com/octonato/minimal-lagom-without-cluster

Check the tests. It isn't using cluster and the loaded config shows it is a local provider.

Please, verify if your loader is not loading any component that is requiring cluster. If the cluster component is included, it will bootstrap the cluster.

Kyle Florence · Answer 3 · Tue Apr 06 2021 00:54:05 GMT+0800 (China Standard Time)

Thanks @octonato -- I will take a closer look at this. Off the top of your head, are you aware of any library dependencies that would cause the cluster to become enabled without any configuration?

Here is the full list of dependencies for that project:

    libraryDependencies ++= Seq(
      Dependencies.akkaHttp,
      Dependencies.jacksonDatabind,
      Dependencies.lagomLogback,
      Dependencies.lagomScaladslClient,
      Dependencies.lagomScaladslTestkit,
      Dependencies.playJson
    ).map(_ % Provided) ++
      build.dependencies.test(IntegrationTest, Provided) ++
      Seq(
        Dependencies.jacksonDatabind,
        Dependencies.lagomScaladslServer,
        Dependencies.lagomScaladslTestkit,
        Dependencies.playAkkaHttpServer
      ).map(_ % IntegrationTest)

And those dependencies:

  val jackson = "2.10.4"
  val akkaHttp = ("com.typesafe.akka" %% "akka-http" % LagomVersion.akkaHttp)
    .exclude("com.fasterxml.jackson.core", "jackson-databind")
  val jacksonDatabind = "com.fasterxml.jackson.core" % "jackson-databind" % Versions.jackson
  val lagomLogback = "com.lightbend.lagom" %% "lagom-logback" % LagomVersion.current
  val lagomScaladslClient = "com.lightbend.lagom" %% "lagom-scaladsl-client" % LagomVersion.current
  val lagomScaladslServer = "com.lightbend.lagom" %% "lagom-scaladsl-server" % LagomVersion.current
  val lagomScaladslTestkit = "com.lightbend.lagom" %% "lagom-scaladsl-testkit" % LagomVersion.current
  val playAkkaHttpServer = "com.typesafe.play" %% "play-akka-http-server" % LagomVersion.play

build.dependencies.test includes ScalaTest:

  val mockitoScala = "1.16.15"
  val scalaTest = "3.2.2"
  val mockitoScala = "org.mockito" %% "mockito-scala" % ScalaProjectVersions.mockitoScala
  val mockitoScalaTest = "org.mockito" %% "mockito-scala-scalatest" % ScalaProjectVersions.mockitoScala
  val scalaTest = "org.scalatest" %% "scalatest" % ScalaProjectVersions.scalaTest

Kyle Florence · Answer 4 · Tue Apr 06 2021 01:17:51 GMT+0800 (China Standard Time)

So, strangely, even in the case where the cluster bootstraps, akka.actor.provider is still set to local 🤔

Added to test:

  val provider = server.actorSystem.settings.config.getString("akka.actor.provider")
  println(">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>")
  println(provider)

Logs:

[INFO ][2021-04-05 12:14:00,437][akka.event.slf4j.Slf4jLogger][application-akka.actor.default-dispatcher-4][] Slf4jLogger started [---]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
local


[INFO ][2021-04-05 12:14:02,783][akka.event.slf4j.Slf4jLogger][application-akka.actor.default-dispatcher-4][] Slf4jLogger started [---]
[INFO ][2021-04-05 12:14:02,940][akka.remote.artery.tcp.ArteryTcpTransport][application-akka.actor.default-dispatcher-4][akkaAddress=akka://application@127.0.0.1:25520, sourceThread=ScalaTest-run-running-ExampleLagomOpenApiValidationSpec, akkaSource=ArteryTcpTransport(akka://application), sourceActorSystem=application, akkaTimestamp=17:14:02.939UTC] Remoting started with transport [Artery tcp]; listening on address [akka://application@127.0.0.1:25520] with UID [-1493401165168292556] [---]
[INFO ][2021-04-05 12:14:02,961][akka.cluster.Cluster][application-akka.actor.default-dispatcher-4][akkaAddress=akka://application@127.0.0.1:25520, sourceThread=ScalaTest-run-running-ExampleLagomOpenApiValidationSpec, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=17:14:02.960UTC] Cluster Node [akka://application@127.0.0.1:25520] - Starting up, Akka version [2.6.8] ... [---]
[INFO ][2021-04-05 12:14:03,158][akka.cluster.Cluster][application-akka.actor.default-dispatcher-4][akkaAddress=akka://application@127.0.0.1:25520, sourceThread=ScalaTest-run-running-ExampleLagomOpenApiValidationSpec, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=17:14:03.157UTC] Cluster Node [akka://application@127.0.0.1:25520] - Registered cluster JMX MBean [akka:type=Cluster] [---]
[INFO ][2021-04-05 12:14:03,158][akka.cluster.Cluster][application-akka.actor.default-dispatcher-4][akkaAddress=akka://application@127.0.0.1:25520, sourceThread=ScalaTest-run-running-ExampleLagomOpenApiValidationSpec, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=17:14:03.158UTC] Cluster Node [akka://application@127.0.0.1:25520] - Started up successfully [---]
[INFO ][2021-04-05 12:14:03,181][akka.cluster.Cluster][application-akka.actor.default-dispatcher-4][akkaAddress=akka://application@127.0.0.1:25520, sourceThread=application-akka.actor.internal-dispatcher-3, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=17:14:03.181UTC] Cluster Node [akka://application@127.0.0.1:25520] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing [---]
[INFO ][2021-04-05 12:14:03,181][akka.cluster.Cluster][application-akka.actor.default-dispatcher-4][akkaAddress=akka://application@127.0.0.1:25520, sourceThread=application-akka.actor.internal-dispatcher-3, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=17:14:03.181UTC] Cluster Node [akka://application@127.0.0.1:25520] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining [---]

I did notice one difference though, my tests include a StandaloneLagomClientFactory, perhaps that is the cause as when I remove it the cluster doesn't seem to start up.

  protected[lagom] lazy val clientFactory: ClientFactory =
    new StandaloneLagomClientFactory(s"OpenApiLagomClient-${UUID.randomUUID}")
      with AhcWSComponents
      with StaticServiceLocatorComponents {
      override def staticServiceUri: URI = validationServer.url.toURI
    }

I did not expect that to start a cluster... For now I can workaround this with:

  protected[lagom] lazy val clientFactory: ClientFactory =
    new StandaloneLagomClientFactory(s"OpenApiLagomClient-${UUID.randomUUID}")
      with AhcWSComponents
      with StaticServiceLocatorComponents {
      override lazy val configuration: Configuration = Configuration.load(
        environment.classLoader,
        System.getProperties,
        Map("akka.actor.provider" -> "local"),
        allowMissingApplicationConf = true
      )

      override def staticServiceUri: URI = validationServer.url.toURI
    }

Kyle Florence · Answer 5 · Tue Apr 06 2021 01:29:40 GMT+0800 (China Standard Time)

So @octonato -- to clarify, the fix for #2042 does work as intended (the server started in the TestKit is properly configured). However, just the inclusion of lagom-scaladsl-testkit seems to, by default, start a cluster for any Lagom application ActorSystem, and since this StandaloneLagomClientFactory is running outside of the TestKit server, it starts up a cluster for itself. I don't find this very intuitive. Actually it seems I ran into this a little over a year ago as well https://gitter.im/lagom/lagom?at=5e013f82cf771f77081c0327

Would be nice if Akka Cluster were opt-in instead of enabled by default, but maybe it's to account for the primary use case of testkit.

Kyle Florence · Answer 6 · Tue Apr 06 2021 01:56:09 GMT+0800 (China Standard Time)

I have updated the description and test case. Thanks for your time as always Renato.

Renato Cavalcanti · Answer 7 · Mon Apr 12 2021 23:32:08 GMT+0800 (China Standard Time)

@solarmosaic-kflorence, sorry for the silence on the line. I was off for a week.

I think what is happening is the following:

The StandaloneLagomClientFactory is intended to be used by applications that do not have their own actor systems, like from any other Java application (no Akka, no Play, no Lagom). It starts its own actor system because it's not expected to have one in scope.

Because it's being started during the tests, it will read the default config and since your application is bringing in all the lagom cluster dependencies, your final default config (defined by the classpath) will have the akka.actor.provider set to cluster.

The Lagom testkit will start the actor system using a specific config, exactly to avoid the prod config. But the StandaloneLagomClientFactory doesn't know about it. Its actor system by default will use the config from the classpath.

So, the first question is, why are you using StandaloneLagomClientFactory from inside Lagom? You don't need it as you can generate a Lagom client from inside a Lagom application. When doing so, you will use the same ActorSystem as your Lagom application (which is recommended) and, in tests, it will use the tweaked config we have for the testkit.

To create a client inside a Lagom application, have a look at Consuming services.

Just for context, have a look at this page where we explain the different integration options you can use. I don't think this is what you need. You need the first link though.

Kyle Florence · Answer 8 · Tue Apr 13 2021 06:23:24 GMT+0800 (China Standard Time)

@octonato It is true that I might be able to fix this by re-configuring my service to use StaticServiceLocatorComponents instead of using StandaloneLagomClientFactory (I will look into that) -- however, I think the core problem here is that I wanted to use the testkit without using Akka Cluster and there doesn't seem to be an easy way to do that, at least without excluding Akka Cluster dependencies when including the testkit. Seems better to have Akka Cluster be an optional dependency. If you disagree, we can close this ticket out.

Renato Cavalcanti · Answer 9 · Tue Apr 13 2021 18:12:35 GMT+0800 (China Standard Time)

I think the core problem here is that I wanted to use the testkit without using Akka Cluster and there doesn't seem to be an easy way to do that, at least without excluding Akka Cluster dependencies when including the testkit

I don't think this is the case. You can have the cluster dependencies in the classpath and the testkit will make sure that its actor system will be configured to use the local provider.

What is happening here is that there is another unmanaged actor system that is being start by the StandaloneLagomClientFactory.

This is not something that I think we should 'fix' because the StandaloneLagomClientFactory wasn't designed to be used in a Lagom application.

Kyle Florence · Answer 10 · Thu Jul 29 2021 09:51:56 GMT+0800 (China Standard Time)

I just ran into this again. I think this arises in cases where we only want to test a single Lagom client without starting up the entire application just to access it, which seems like overkill. I suppose I can just make a very simple LagomApplication that I use to get the client instance instead of a StandaloneLagomClientFactory, although that feels a bit hacky.

trait LagomClient extends Service {
  override def descriptor: Descriptor = Service.named("lagom-client")
}

class LagomClientImpl extends LagomClient

abstract class LagomClientApplication(context: LagomApplicationContext)
  extends LagomApplication(context)
  with AhcWSComponents {
  lazy val lagomServer: LagomServer = serverFor[LagomClient](new LagomClientImpl)
}

lazy val server = ServiceTest.startServer(ServiceTest.defaultSetup) { context =>
  new LagomClientApplication(context) with StaticServiceLocatorComponents {
    override def staticServiceUri: URI = URI.create(mockServer.getUrl)
  }
}

// Get the client...
lazy val client = server.serviceClient.implement[Whatever]

But at least it will respect the configuration that I want.

Renato Cavalcanti · Answer 11 · Mon Aug 23 2021 18:47:46 GMT+0800 (China Standard Time)

Yes, that sounds hacky.
A better option is to start an actor system using a different config. Look for overloaded constructors.

Kyle Florence · Answer 12 · Tue Aug 24 2021 01:00:17 GMT+0800 (China Standard Time)

@octonato but that will get us back to the original issue, where starting a separate actor system (outside of the Lagom application, for example with StandaloneLagomClientFactory) will use the wrong configuration. I think there is no non-hacky way to do this as long as the testkit is included.