This repository serves as a demonstration how to treat OOMKilled error for a stateful Kafka Streams applications running in Kubernetes
I used kafka connect datagen connector (link to docs).
To generate load I used config in file datagen_connector_config.json and schema in test_topic.avsc.
- Uncomment KafkaStreams configuration in Application.java:43
- Uncomment heap limits in deployment.yaml:43
| Metric | PromQL |
|---|---|
| JVM memory used | sum by (pod) (jvm_memory_used_bytes{pod=”your-pod-name"}) |
| Container memory usage | container_memory_working_set_bytes{pod="your-pod-name",container="app"} |
| Container memory limit | container_spec_memory_limit_bytes{pod="your-pod-name",container="app"} |
| RocksDB memory usage + os page cache | kafka_stream_state_size_all_mem_tables{pod=~"your-pod-name"} + kafka_stream_state_block_cache_usage{pod=~"your-pod-name"} + kafka_stream_state_estimate_table_readers_mem{pod=~"your-pod-name"} + on () container_memory_cache{pod="your-pod-name”, container="your-container-name"} |
- Create a local .properties file with kafka streams properties. For example:
application.id: kafka-streams-oom-killed
state.dir: /tmp/states
bootstrap.servers: localhost:9092
schema.registry.url: http://localhost:8081- Add env var CONFIG_FILE_PATH with a path to the properties file.
To deploy to k8s you'll need a running k8s cluster and docker, helm installed on your local machine.
- build shadow jar
./gradlew shadowJar - build Docker image
docker build -t your-image-tag:0.0.1 .- push Docker image to your preferred registry
docker push your-image-tag:0.0.1- change image in deployment.yaml
- add kafka streams configuration to helm/templates/configmap.yaml
- execute helm update --install
helm upgrade --install --atomic --namespace your-namespace your-release-name helm/