This repository describes the steps and resources to deploy Kafka on Kubernetes using Strimzi.
** Deploy the Strimzi operator and then the Kafka cluster resources.**(Use resources from kafka-demo/demo1-strimzi-setup path)
1- Starting Minikube:
minikube start --memory=4096
2- Create Kafka namespace:
kubectl create namespace kafka
3- Deploy Strimzi Operator (including ClusterRole, ClusterRoleBinding and CRDs):
kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka
4- Now we can add a Kafka custom resource to deploy a Kafka Cluster:
kubectl apply -f kafka.yaml -n kafka
5- Wait for the Kafka, Zookeeper and other optional resources to be ready
6- Create a Topic:
kubectl apply -f kafka-topic.yaml -n kafka
7- Run Producer
kubectl -n kafka run kafka-producer -ti --image=quay.io/strimzi/kafka:0.28.0-kafka-3.1.0 --rm=true --restart=Never -- bin/kafka-console-producer.sh --bootstrap-server kafka-kafka-bootstrap:9092 --topic my-topic
8- Run Consumer
kubectl -n kafka run kafka-consumer -ti --image=quay.io/strimzi/kafka:0.28.0-kafka-3.1.0 --rm=true --restart=Never -- bin/kafka-console-consumer.sh --bootstrap-server kafka-kafka-bootstrap:9092 --topic my-topic --from-beginning
Demo-3: This demo is to show the Kafka cluster monitoring using Prometheus and GrafanaUse resources from kafka-demo/demo-3 monitoring path)
1- Create monitoring namespace:
kubectl create namespace monitoring
2- Deploy Prometheus operator:
kubectl apply -f prometheus-operator-deployment.yaml -n monitoring --force-conflicts=true --server-side
3- Create configmap for jmx metrics:
kubectl apply -f kafka-metrics-config.yaml -n kafka
kubectl apply -f zookeeper-metrics.yaml -n kafka
4- Update the Kafka resource with jmxPrometheusExporter to scrape the jmx metrics and kafkaExporter for exporting the topic and consumer lag metrics
kubectl apply -f kafka.yaml -n kafka
5- Deploy Prometheus -
kubectl apply -f prometheus.yaml -n monitoring
6- Deploy pod monitor-
kubectl apply -f strimzi-pod-monitor.yaml -n monitoring
7- Deploy Grafana -
kubectl apply -f grafana.yaml -n monitoring
8- Port forward for Prometheus and Grafana-
kubectl port-forward svc/grafana 3000:3000 -n monitoring
kubectl port-forward svc/prometheus-operated 9090:9090 -n monitoring
9- Add Prometheus datasource in Grafana 10 Upload grafana dashboards from- https://github.com/strimzi/strimzi-kafka-operator/tree/0.28.0/examples/metrics/grafana-dashboards