Ask for help testing a topology with multiple consumers on multiple output topics.

[mathieu]

Hello,
This is more a request for help than reporting an issue. I have a strange unpredictable behavior I struggle to understand.
I have a topology with several input and output topics. In unit tests, everything's fine.
My integration tests, using EmbeddedKafka, looks roughly like this:

withRunningKafka {

  createCustomTopic(topic = "in1", partitions = numParts)
  createCustomTopic(topic = "in2", partitions = numParts)

  val streams = new KafkaStreams(topology,  ...config... )
  
  publishToKafka("in1", List(...))
  publishToKafka("in2", List(...))

  streams.start()      // yes, I want to start the stream after feeding the topics (as it may happen IRL), as I want to check the proper management of timestamps among others...
   
  withConsumer[...] { 
      consumer =>
        val events = consumer.consumeLazily[...]("out1").take(10).toList
        events.size shouldBe 6
  }

  withConsumer[...] { 
      consumer =>
        val events = consumer.consumeLazily[...]("out2").take(10).toList
        events.size shouldBe 6
  }
 
  withConsumer[...] { 
      consumer =>
        val events = consumer.consumeLazily[...]("out3").take(30).toList
        events.size shouldBe 24
  }

  streams.close() // actually in a finally block
}

Now, I don't have consistent results : if I swap the 2 last consumer blocks, some tests are failing (0 events fetched), or not. The first block intermittently fails.
Is there anything I'm doing wrong in this setup, or with the consumers ?

Also, depending on the number of partitions numParts on inputs, I don't have the same results. You would say this is a problem in my topology, but so far I didn't find any reason for this.

I'd be glad if you have any hints or suggestion on these. Thanks

[mathieu]

Strangely, using the consumeNumberMessagesFrom() API, I'm back to a predictable behavior. I don't get why though.