rbiedrawa / kafka-streams-state-store-monitoring-poc

This repository demonstrates how to measure StateStore restoration process time inside Kafka Streams application.

Kafka Streams StateStore restoration process monitoring (PoC)

This repository demonstrates how to measure StateStore restoration process time inside Kafka Streams application.

Features

• Custom StateRestoreListener for gathering metrics.
• Zipkin integration for metrics visualization.

Getting Started

Prerequisites

• Java 8
• Docker

Usage

• Build docker image.

  ./gradlew bootBuildImage

• Start docker-compose demo.

    docker-compose up -d

• List containers and check if all are Up.

  docker-compose ps   
  #                    Name                                  Command                  State                                      Ports                               
  # -----------------------------------------------------------------------------------------------------------------------------------------------------------------
  # kafka-statestore-monitoring_fast-data-dev_1   /usr/bin/dumb-init -- /usr ...   Up             0.0.0.0:8081->8081/tcp, 0.0.0.0:8082->8082/tcp, 3031/tcp,                                                                    
  # kafka-statestore-monitoring_zipkin_1          start-zipkin                     Up (healthy)   9410/tcp, 0.0.0.0:9411->9411/tcp                                   
  # kafka-stream-demo-app                         /cnb/process/web                 Up             0.0.0.0:8080->8080/tcp                                             

• Follow logs of demo-app.

  docker-compose logs -f kafka-stream-demo-app

• Produce dummy messages to add some data to state store.

  # start kafka console producer
  docker-compose exec fast-data-dev kafka-console-producer --broker-list fast-data-dev:9092 --topic words
  
  # sample logs from 'demo-app'
  # kafka-stream-demo-app    | 2021-04-10 14:17:18.708  INFO [hello-world,,] 1 --- [-StreamThread-1] c.rbiedrawa.app.words.WordCountKStream   : Received word: hello world
  # kafka-stream-demo-app    | 2021-04-10 14:17:21.312  INFO [hello-world,,] 1 --- [-StreamThread-1] c.rbiedrawa.app.words.WordCountKStream   : Received word: test test1

• Restart demo-app to trigger state store restoration.

  docker-compose restart kafka-stream-demo-app

• View demo-app logs, StateRestoreListener should print some metrics, below sample output:

  kafka-stream-demo-app    | 2021-04-10 14:21:00.603  INFO [hello-world,,] 1 --- [-StreamThread-1] c.r.a.kafka.ZipkinStateRestoreListener   : Started restoration of WordCounts partition 0 total records to be restored 6
  kafka-stream-demo-app    | 2021-04-10 14:21:00.604  INFO [hello-world,,] 1 --- [-StreamThread-1] c.r.a.kafka.ZipkinStateRestoreListener   : Started restoration of WordCounts partition 5 total records to be restored 1
  kafka-stream-demo-app    | 2021-04-10 14:21:00.854  INFO [hello-world,,] 1 --- [-StreamThread-1] c.r.a.kafka.ZipkinStateRestoreListener   : Restored batch 1 for WordCounts partition 5 
  kafka-stream-demo-app    | 2021-04-10 14:21:00.856  INFO [hello-world,,] 1 --- [-StreamThread-1] c.r.a.kafka.ZipkinStateRestoreListener   : Restoration complete for WordCounts partition 5 totalRestoredRecords: 1 totalTime: 251
  kafka-stream-demo-app    | 2021-04-10 14:21:01.011  INFO [hello-world,,] 1 --- [-StreamThread-1] c.r.a.kafka.ZipkinStateRestoreListener   : Restoration complete for WordCounts partition 0 totalRestoredRecords: 6 totalTime: 406

• Open your web browser and go to Zipkin dashboard page.

• Use tagQuery=state-restore to filter only state-restore traces.

• Click SHOW button to see more details.

• Stop docker compose.

  docker-compose down -v

References

• StateRestoreListener

License

Distributed under the MIT License. See LICENSE for more information.