j-tim / conduktor-playground

Playground for Conduktor - Kafka

Conduktor Playground

Docker

Basic setup: docker-compose.yml

Component	Port	API / UI	Description	Documentation
Kafka broker	9092		Single Node Kafka Cluster	https://kafka.apache.org/documentation/
Conduktor Plaform	80	Conduktor	Conduktor	https://docs.conduktor.io/platform/
Confluent Schema Registry	8081	Schema Registry	Schema Registry for Avro Schemas	https://docs.confluent.io/platform/current/schema-registry/index.html

Kafka connect: docker-compose-kafka-connect.yml

Component	Port	API / UI	Description	Documentation
Nginx	8083	Nginx	Loadbalancer for Kafka connect cluster	https://nginx.org/en/docs/
Kafka Connect - Worker 1	18083	Kafka Connect - Worker 1	3 worker nodes cluster	https://docs.confluent.io/platform/current/connect/index.html
Kafka Connect - Worker 2	28083	Kafka Connect - Worker 2	"	https://docs.confluent.io/platform/current/connect/index.html
Kafka Connect - Worker 3	38083	Kafka Connect - Worker 3	"	https://docs.confluent.io/platform/current/connect/index.html

Conduktor Proxy: docker-compose-conduktor-proxy.yml

Component	Port	Browser	Description	Documentation
Conduktor Proxy	8888, 6969 - 6970	Conduktor Proxy API	Proxy in front of Kafka cluster	https://docs.conduktor.io/platform/proxy

Kafka console producer / consumer docker-compose-console-consumer-producer.yml

Component	Port	Description	Documentation
Kafka broker used as console producer	none	Container used only for the console producer	https://developer.confluent.io/tutorials/kafka-console-consumer-producer-basics/confluent.html
Kafka broker used as console consumer	none	Container used only for the console consumer	https://developer.confluent.io/tutorials/kafka-console-consumer-producer-basics/confluent.html

Single node Kafka cluster (including Conduktor platform)

Up:

docker-compose up -d

Down:

docker-compose down -v

Conduktor

• admin:
  • username: [e-mail]
  • password: admin

With Kafka Connect

Up:

docker-compose -f docker-compose.yml -f docker-compose-kafka-connect.yml up -d 

Down:

docker-compose -f docker-compose.yml -f docker-compose-kafka-connect.yml down -v

Kafka Connect

Up:

docker-compose -f docker-compose.yml -f docker-compose-kafka-connect.yml up -d

Down:

docker-compose -f docker-compose.yml -f docker-compose-kafka-connect.yml down -v

Start generating random data using Kafka Connect Data Gen:

IntelliJ Http Client - Kafka Connect API calls

or using curl:

Create connector:

curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d '{
  "name": "datagen-stock-quotes",
  "config": {
    "connector.class": "io.confluent.kafka.connect.datagen.DatagenConnector",
    "kafka.topic": "stock-quotes",
    "value.converter.schemas.enable": "true",
    "schema.filename": "/kafka-connect/datagen/avro/StockQuote.avsc",
    "schema.keyfield": "symbol",
    "max.interval": 10,
    "iterations": -1,
    "tasks.max": 1
  }
}'

Update configuration:

curl -X POST http://localhost:8083/connectors/datagen-stock-quotes/config -H "Content-Type: application/json" -d '{
  "connector.class": "io.confluent.kafka.connect.datagen.DatagenConnector",
  "kafka.topic": "stock-quotes",
  "value.converter.schemas.enable": "true",
  "schema.filename": "/kafka-connect/datagen/avro/StockQuote.avsc",
  "schema.keyfield": "symbol",
  "max.interval": 10,
  "iterations": -1,
  "tasks.max": 3
}'

Conduktor proxy

Up:

docker-compose -f docker-compose.yml -f docker-compose-kafka-connect.yml -f docker-compose-conduktor-proxy.yml up -d

Down:

docker-compose -f docker-compose.yml -f docker-compose-kafka-connect.yml -f docker-compose-conduktor-proxy.yml down -v

Console producer & consumer

Up:

docker-compose -f docker-compose.yml -f docker-compose-kafka-connect.yml -f docker-compose-conduktor-proxy.yml -f docker-compose-console-consumer-producer.yml up -d

Down:

docker-compose -f docker-compose.yml -f docker-compose-kafka-connect.yml -f docker-compose-conduktor-proxy.yml -f docker-compose-console-consumer-producer.yml down -v

Create topic:

docker exec kafka-console-producer \
  kafka-topics \
    --bootstrap-server conduktor-proxy:6969 \
    --create --if-not-exists --topic conduktor-test-topic

List the topics:

docker-compose exec kafka-console-producer \
  kafka-topics \
    --bootstrap-server conduktor-proxy:6969 \
    --list

Producer to the topic:

echo 'Hello World' | docker-compose exec -T kafka-console-producer \
    kafka-console-producer  \
        --bootstrap-server conduktor-proxy:6969 \
        --topic conduktor-test-topic

Consume from the topic:

docker-compose exec kafka-console-consumer kafka-console-consumer \
  --bootstrap-server conduktor-proxy:6969 \
  --from-beginning \
  --topic conduktor-test-topic

Build the project

./mvnw clean install

Run the consumer application

./mvnw spring-boot:run -pl spring-kafka-consumer

Example Repos

• https://github.com/conduktor/kafka-stack-docker-compose
• https://github.com/conduktor/conduktor-platform/
• https://github.com/conduktor/conduktor-platform/blob/main/example-local/
• https://github.com/conduktor/conduktor-proxy-demos/tree/main/chaos

Documentation

• https://www.conduktor.io/kafka/how-to-start-kafka-using-docker
• Documentation: https://docs.conduktor.io/platform/

Docker

• https://github.com/conduktor/conduktor-platform/blob/main/doc/Configuration.md
• https://registry.hub.docker.com/r/conduktor/conduktor-platform/tags

Configuration

https://github.com/conduktor/conduktor-platform/blob/729d6521a785d116dafab20fcffc18abed47c87a/doc/Configuration.md

Changelog / Release notes

Conduktor - Changelog