naren-jha / KafkaDemo

Kafka demo application using spring-kafka

KafkaDemo

Resources

• https://www.madhur.co.in/blog/2020/08/30/kafka-consumer-multithreading.html#:~:text=In%20the%20multi%2Dthreaded%20consumer,them%20to%20process%20in%20parallel.
• https://docs.google.com/document/d/1kFx9WSU86ivEemsnacM1y4yG6cCNeIeFxiBYjJ5pvVU/edit
• https://youtu.be/SqVfCyfCJqw
• https://www.baeldung.com/spring-kafka
• https://youtu.be/R873BlNVUB4
• https://youtu.be/1vLMuWsfMcA
• https://youtu.be/JalUUBKdcA0
• https://youtu.be/UEg40Te8pnE
• https://youtu.be/STKCRSUsyP0

Stackoverflow -

• https://stackoverflow.com/search?q=user:1240763+[spring-kafka]
• Listeners and concurrency:
• https://stackoverflow.com/a/68978435/4210068
• https://stackoverflow.com/a/55033589/4210068
• https://stackoverflow.com/q/48254650/4210068
• https://stackoverflow.com/a/64012713/4210068
• https://stackoverflow.com/a/50254280/4210068
• https://stackoverflow.com/a/60837978/4210068
• https://stackoverflow.com/questions/23751708/is-zookeeper-a-must-for-kafka

Docs

• https://docs.spring.io/spring-kafka/docs/current/reference/html/#receiving-messages
• https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#messaging.kafka.receiving
• https://kafka.apache.org/documentation/#producerconfigs
• https://kafka.apache.org/documentation/#consumerconfigs

Testing

Curl

curl --location --request POST 'localhost:8081/user/pushToKafka' \
--header 'Content-Type: application/json' \
--data-raw '{
    "id": 10000,
    "name": "Katie Jacobs",
    "age": 35
}'

Verify data is pushed correctly

Verify data reception by consumer

SETUP

Kafka, Zookeeper, Docker setup

first check if docker is installed docker -v If not, then install docker first brew install --cask docker

then create a file called docker-compose.yml in user home directory and paste following content into it

version: "3"
services:
  zookeeper:
	image: 'bitnami/zookeeper:latest'
	container_name: zookeeper
	ports:
  	- '2181:2181'
	environment:
  	- ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
	image: 'bitnami/kafka:latest'
	container_name: kakfa
	ports:
  	- '9092:9092'
	environment:
  	- KAFKA_BROKER_ID=1
  	- KAFKA_LISTENERS=PLAINTEXT://:9092
  	- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
  	- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
  	- ALLOW_PLAINTEXT_LISTENER=yes
	depends_on:
  	- zookeeper

then run

docker-compose -f docker-compose.yml up docker compose up

docker compose up -d

-d(for detach)

or run using docker desktop client