DivLoic / xke-ratatouille

:pill:Poison pills and Kafka Streams demo:curry::rat:

XKE Ratatouille: Streaming App and Poison Pills

This module illustrates the concept of poison pill 💊 for streaming applications and shows how to face this issue with Kafka Streams, a client library for building continuous processing applications with Apache Kafka.

You might experience that some of the incoming records from your Kafka Streams application are corrupted, or that the serializer/deserializer might be incorrect or buggy, or cannot handle all record types. These types of records are referred to as “poison pills”. -- Confluent FAQ

tl;dr

The module is also the support for a few talks, here are the two major ones:

• Devoxx France 2019: Comment Kafka Streams compte faire passer la pillule_🇫🇷
• Kafka Summit San Francisco 2019: handle the unexpected with Kafka Streams

Table of contents

• Storytelling
• Run the full demo
  • Setup
  • Sentinel Value
  • Dead Letter Queue
  • Clean up
• Complete the exercises TODO

Perquisite

• hexdump
• Docker
• jdk >= 1.8

Storytelling

Really old systems receive raw bytes directly from message queues. With Kafka (Connect and Streams) we’d like to continuously transform these messages. But we need a deserializer with special decoder to understand each event. What happens if we get a buggy implementation of the deserializer?

Run the full demo

Setup

Checkout the master branch and open 3 terminals, and start docker.

$ git clone https://github.com/DivLoic/xke-ratatouille.git
$ cd xke-ratatouille

Run the gradlew-compose script. This will, compile the project, create docker images and run them.

$ ./gradlew-compose.sh

# expected output
#
#        Name                      Command               State                       Ports
# -------------------------------------------------------------------------------------------------------------
# elasticsearch         /usr/local/bin/docker-entr ...   Up      0.0.0.0:9200->9200/tcp, 9300/tcp
# grafana               /run.sh                          Up      0.0.0.0:3000->3000/tcp
# kafka                 /etc/confluent/docker/run        Up      0.0.0.0:9092->9092/tcp, 0.0.0.0:9094->9094/tcp
# kafka-connect         /etc/confluent/docker/run        Up      0.0.0.0:8083->8083/tcp, 9092/tcp
# kibana                /usr/local/bin/kibana-docker     Up      0.0.0.0:5601->5601/tcp
# prometheus            /bin/prometheus --config.f ...   Up      0.0.0.0:9090->9090/tcp
# ratatouille-app       /bin/sh -c http-server .         Up      0.0.0.0:8080->8080/tcp
# ratatouille-datagen   /opt/ratatouille-datagen/b ...   Up      0.0.0.0:7070->7070/tcp
# schema-registry       /etc/confluent/docker/run        Up      0.0.0.0:8081->8081/tcp

In a new terminal, run the consumer script.

$ ./consume-input.sh

Open a web browser at the address: http://localhost:8080.
An interface with four toggles lets you generate records. Among theses records some are corrupted, that's the poison pills 💊 specially introduced by the generator.

In a new terminal, start the streaming app.

$ ./gradlew demo

Since it's the correction, this version take care of all the corrupted messages.

Sentinel Value

Go on http://localhost:3000/d/YmOgiAgWk/ (admin/admin🤫). You should see, in the middle of a dashboard, the number of corrupted elements.

Switch off the exercise 3 to see the number off Sentinel Value decrease.
Culprit spoted🎉! Some drinks are not serialised as expected. Actually this is the case for all the wines.

Dead Letter Queue

Scroll down the interface and switch on the Elasticsearch connector. This connector send messages from the dlq-food-error topic to a Elasticsearch index. Then you can visualise them in Kibana.

All corrupted messages in dinner flow end with 414554.

414554 -> AET -> Australia/Sydney

Culprit spoted🎉! Corrupted messages are sent by a brand new restaurant in Australia.

Clean up

Stop all your running commands (Ctrl+C). Stop and destroy all containers:

$ docker-compose down

Complete the exercises

TODO