Kafka Kraft is a project that provides a setup for a multi-node Apache Kafka cluster using Docker Compose.
Ensure Docker and Docker Compose are installed on your machine.
-
Clone the repository:
git clone https://github.com/daalvand/kafka-kraft.git cd kafka-kraft
-
Duplicate the environment variable file:
cp .env.example .env
-
Modify the
.env
file with your preferred configurations. -
Execute the startup script:
./start.sh
This script verifies the existence of the
.env
file, duplicates it from.env.example
if not found, and starts the Kafka cluster using Docker Compose. If theCLUSTER_ID
is not set in the.env
file, it generates a new cluster ID.
The primary Docker Compose configuration file for the multi-node Kafka cluster. It outlines three Kafka broker services (kafka1
, kafka2
, kafka3
) with unique hostnames, ports, and environment variables.
- Tweak the environment variables in the
.env
file to meet your needs. - Alter the Docker Compose file (
docker-compose.yml
) to modify the Kafka cluster configuration.
To initiate the Kafka cluster:
docker compose up -d
To halt the Kafka cluster:
docker compose down
Ensure Kafka is running. Use the following command to create a Kafka topic named first_topic
with 4 partitions and a replication factor of 2:
docker compose exec kafka1 kafka-topics --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --create --replication-factor 2 --partitions 4 --topic first_topic
docker compose exec kafka1 kafka-topics --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --describe --topic first_topic
You can use the kafka-console-producer
and kafka-console-consumer
scripts to produce and consume data.
Here's an example command to produce data to the first_topic
topic:
docker compose exec kafka1 kafka-console-producer --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --topic first_topic
You can then type your messages into the console, and they will be sent to the first_topic
topic.
Here's an example command to consume data from the first_topic
topic:
docker compose exec kafka1 kafka-console-consumer --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --topic first_topic --from-beginning
with consumer group:
docker compose exec kafka1 kafka-console-consumer --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --topic first_topic --group=my_first_group
This command will consume and print all messages from the first_topic
topic.
- This project utilizes the official Confluent Kafka Docker image version 7.5.2.
- The
start.sh
script automates the setup process and verifies necessary configurations.