suadev / microservices-change-data-capture-with-debezium

Microservices data exchange with change data capture and outbox pattern.

This simple project demonstrates how to manage eventual consistency between microservices with Change Data Capture and Outbox Pattern using Debezium, Kafka, and Kafka Connect.

Prerequisites

• .NET 5.0 SDK
• Docker Desktop

Run in Debug Mode

• Run 'docker-compose up' and wait for all infra to up and running.
• Select 'All' debug option and start debugging. (for vs code)
• Wait until all microservices are up and running.

Initiating Databases: Each service will be created its own database while it's starting for the first time.

Register Debezium Postgres Connectors to Kafka Connect

You need to register two Postgres Connectors. One for Customer Database and the other for Identity Database.

customer_config.json -> Customer Connector Config.

identity_config.json -> Identity Connector Config.

Use customer.sh and identity.sh to create/update/delete connectors. For instance, to create customer connector;

.\customer.sh create_connector

update_connector function is commented out. If you want to update the connector config, uncomment the function and download jq from here.

After registeration of two Debezium Connectors, two workers will be created on Kafka Connect which are listening to the outbox tables to push events to Kafka topics.

Check the connector list via the following endpoint and see the following json result to be sure everything is okay.

curl -X GET http://localhost:8083/connectors 

["identity_outbox_connector", "customer_outbox_connector"]

Now you are ready to test. See sample postman requests here.

Overall Architecture

When a new user created on Identity Service, eventual consistency will be obtained for Customer and Notification Services as shown following flow.

Tool Set

• Asp.Net 5.0
• Entity Framework Core 5.0
• PostgreSQL - Npgsql
• MediatR
• Kafka - Zookeeper
• Confluent.Kafka
• Kafka Connect
• Debezium
• Kafdrop
• Docker - Docker Compose
• Azure Data Studio
• VS Code