Version 2.4.1 giving topic "not present in metadata after 60000 ms"

[Kevin Breidenbach]

After upgrading to embedded-kafka 2.4.1 (as well as upgrading the kafka client to the same version) we are seeing error such as:

Topic anytopic not present in metadata after 60000 ms

If we keep the kafka client at 2.4.1 and switch embedded-kafka back to 2.4.0 we do not see the error.

We are using withRunningKafka to start embedded kafka, but have tried to explicitly start/stop to no avail. We are using default configuration.

Scala version: 2.12.8
SBT version: 1.3.9
Kafka client version: 2.4.1
embedded kafka version: 2.4.1

[Francesco Pellegrini]

Hi @kbreidenbach, can you please provide a code snippet of your test?

[Kevin Breidenbach]

I can't share work code, but I'll try to put something together this evening.

The problem appears to be that the code we're testing uses their own Kafka producers/consumers rather than those created with EmbeddedKafka.withConsumer/Producer.

Basically, in order to use v 2.4.1 onward we would need to refactor our code so we pass in the consumers and producers that we want to test with, which is not convenient. We may just switch to using docker based Kafka as we can test with that without changing our codebase

[Kevin Breidenbach]

This is by no means a good test, but shows the problem we have. If our apps create their own producers or consumers then we cannot test using embedded-kafka v 2.4.1 as we get errors as shown in the comments of the code snippet below.

We cannot rewrite all our apps to accept consumers and producers through DI and have tests pass in using EmbeddedKafka.withConsumer/Producer. So we were hoping you could fix this so we're not stuck using v 2.4.0.

import java.util.Properties

import net.manub.embeddedkafka.EmbeddedKafka
import org.apache.kafka.clients.producer.{KafkaProducer, ProducerConfig, ProducerRecord}
import org.apache.kafka.common.serialization.StringSerializer
import org.scalatest.freespec.AnyFreeSpec
import org.scalatest.matchers.should.Matchers

class KafkaTest extends AnyFreeSpec with Matchers with EmbeddedKafka  {

  "given a custom producer we get an error" in withRunningKafka {
    val properties = new Properties()
    properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:6001")

    val producer = new KafkaProducer(properties, new StringSerializer, new StringSerializer)
    val topic = "test"

    // This line will produce:
    // org.apache.kafka.common.errors.TimeoutException: Topic test not present in metadata after 60000 ms
    val metaData = producer.send(new ProducerRecord[String, String](topic, "Test Message")).get
    metaData.topic() shouldBe topic
  }

}

[Francesco Pellegrini]

Found it: in 47f54c9 Kafka and Zookeeper default ports have been swapped, so now Kafka listens on port 6000 instead of 6001.

You can either:

• update the port in your tests, or
• provide an implicit EmbeddedKafkaConfig from which you'd read the actual port instead of hard-coding it in the Properties. :)

Sorry about this, I might consider releasing a patched 2.4.1.1 version (and update embedded-kafka-schema-registry as well)...

[Kevin Breidenbach]

Thanks for finding that so quickly!