mjstrasser / cdc-kafka-streams

Example CDC Kafka Streams code to accompany Confluent blog article

CDC and Kafka Streams example code

This repository contains code that illustrates the technique described in Combining CDC Transactional Messages Using Kafka Streams. It is a completely new code base derived from that used in the client work described in that article. It uses CDC message formats based on those used in that client work.

The topology

The Kafka Streams topology is defined in TopologyBuilder.kt. It is tested using the TopologyTestDriver with current test results published here.

This example does not include code for detecting incomplete transactions and controlling state store size.

Avro serialization

All messages have string-serialized keys and Apache Avro-serialized values. In all cases, Avro schema definitions were written manually (see src/main/avro directory) and Java classes generated from those definitions

For example:

• transaction.avsc defines the Transaction class.
• The Gradle task generateAvroJava generates Transaction.java that is used in the code.
• Kafka Streams serializes Transaction objects for storing in the state store and deserializes them when needed.

Other code

Supporting code includes:

• Kotlin type aliases that make working with CDC message classes easier.
• Test helpers that make writing clear tests simpler.

Other libraries

• Testing uses the amazing and flexible Kotest framework for Kotlin. (This example code does not include unit tests; only topology tests.)
• Test reports are generated using Kotest HTML Reporter.
• Logging uses Klogging framework for Kotlin.