Kafka Command line is needed to install these commands Run inside the bin directory of kafka
-
Create a new topic:
bash ./kafka-topics.sh --create --bootstrap-server kafka-ip:9092 --replication-factor 1 --partitions 500 --topic topicName
-
Increase number of partitions
bash ./kafka-topics.sh --bootstrap-server kafka-ip:9092 --alter --topic topicName --partitions 6000
-
Describe Topic - Get info about topic
bash ./kafka-topics.sh --describe --zookeeper kafka-ip:2181 --topic topicName
-
Delete Topics
bash ./kafka-run-class.sh kafka.admin.TopicCommand --delete --topic topicName --bootstrap-server kafka-ip:2181
-
Producer - Enter command wait for '>' to appear, type/paste message to publish and press Enter
bash ./kafka-console-producer.sh --broker-list kafka-ip:9092 --topic topicName
-
Consumer - Starts from the latest message ie message being published after starting consumer
bash ./kafka-console-consumer.sh --bootstrap-server kafka-ip:9092 --topic topicName
Consumer - Fetch messages from beginning ie all available messages and keep consuming indefinately
bash ./kafka-console-consumer.sh --bootstrap-server kafka-ip:9092 --topic topicName --from-beginning
Consumer - Fetch messages from beginning and stop after consuming N messages
bash ./kafka-console-consumer.sh --bootstrap-server kafka-ip:9092 --topic topicName --from-beginning --max-messages=N
-
Get Lag - Command will generate a big table like structure containing information about cosumers including their IPs and lag on each partition
bash ./kafka-consumer-groups.sh --bootstrap-server kafka-ip:9092 --describe --group <group-id>
Get Lag - Command will generate a single number, containing sum of lag on all partitions
bash ./kafka-consumer-groups.sh --bootstrap-server kafka-ip:9092 --describe --group <group-name> | grep -E "LAG|*" | awk '{SUM += $6} END { print SUM}'
-
Reset offset for a particular consumer group - If you want to go to the start of messages on a topic for a particular consumer group(can replace 0 with any offset number) :
bash ./kafka-consumer-groups.sh --bootstrap-server kafka-ip:9092 --group group-name --reset-offsets --to-offset 0 --topic topic-name --execute
Reset offset for a particular consumer group - If you want to go to the end of messages on a topic for a particular consumer group(latest messages) :
bash ./kafka-consumer-groups.sh --bootstrap-server kafka-ip:9092 --group group-name --reset-offsets --to-latest --topic topic-name --execute
-
Delete a consumer group
bash ./kafka-consumer-groups.sh --bootstrap-server kafka-ip:9092 --delete --group group-name