[Feature]: Kafka (KRaft) - customizable startup script/advertised listeners

[Chris Roush]

Problem

When building a network with multiple containers that depend on the Kafka container, the startup script assumes that connections are only being attempted between the host and the Kafka cluster directly. Because the startup script has a hard-coded KAFKA_ADVERTISED_LISTENERS environment variable exported, other dependent services (such as Debezium) are not able to communicate with the container because the advertised listeners are only accessible from the host and are not directly reachable in the container network.

My simple usecase setup is as follows:

1. Create a Kafka container and give it an alias
2. Create a Debezium container and have it connect to the Kafka container

Here is the sample Debezium container setup that I am building, using a generic container request

debeziumContainer, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
	ContainerRequest: testcontainers.ContainerRequest{
		Image:          "debezium/connect:1.6",
		ExposedPorts:   []string{"8083"},
		WaitingFor:     wait.ForLog("Finished starting connectors and tasks").WithStartupTimeout(30 * time.Second),
		Networks:       network,
		NetworkAliases: aliases,
		Env: map[string]string{
			"BOOTSTRAP_SERVERS":                      "kafka:9093",
			"GROUP_ID":                               "1",
			"CONFIG_STORAGE_TOPIC":                   "debezium_connect_config",
			"OFFSET_STORAGE_TOPIC":                   "debezium_connect_offsets",
			"STATUS_STORAGE_TOPIC":                   "debezium_connect_status",
			"CONNECT_KEY_CONVERTER_SCHEMAS_ENABLE":   "false",
			"CONNECT_VALUE_CONVERTER_SCHEMAS_ENABLE": "false",
		},
	},
	Started: true,
})

Solution

There are a few ways this could potentially be handled.

1. Provide a customization option for the advertised listeners environment variable that appends to the existing listeners list.

kafkaContainer, err := kafka.RunContainer(ctx, "confluentinc/confluent-local:7.5.0", 
   kafka.WithClusterID("test-cluster"),
   kafka.WithAdvertisedListeners([]string{"PLAINTEXT://kafka:9093"}), // append to existing environment variable defaults
   network.WithNewNetwork(ctx, []string{"kafka"}, network.WithLabels(map[string]string{
       "name": "kafka",
   }), network.WithDriver("bridge")),
)

2. Let testcontainers.WithEnv() take precedence over the exported environment variables in the startup script:

kafkaContainer, err := kafka.RunContainer(ctx, "confluentinc/confluent-local:7.5.0", 
   kafka.WithClusterID("test-cluster"),
   testcontainers.WithEnv(map[string]string{
	"ADVERTISED_HOST_NAME":       "kafka",
	"KAFKA_ADVERTISED_LISTENERS": "PLAINTEXT://kafka:9093,BROKER://kafka:9092",
   }),
   network.WithNewNetwork(ctx, []string{"kafka"}, network.WithLabels(map[string]string{
       "name": "kafka",
   }), network.WithDriver("bridge")),
)

Benefit

The benefit of this feature gives more flexibility in customizing the Kafka cluster for inter-container connectivity. Nothing would change with default behavior, but having more control over advertised listeners of the Kafka cluster will allow for more complex configurations.

It would allow Go-based alternatives to the officially-supported Java Debezium test container: https://github.com/debezium/debezium/blob/main/debezium-testing/debezium-testing-testcontainers/src/main/java/io/debezium/testing/testcontainers/DebeziumContainer.java

Alternatives

Environment variable precedence or an additional option made the most sense to me, but I am happy to talk about alternatives if the maintainers have opinions on certain design principals.

Would you like to help contributing this feature?

Yes