vagrant-kafka
Vagrant config to setup a partitioned Apache Kafka installation with clustered Apache Zookeeper
This configuration will start and provision 6 CentOS6 VMs:
- Three Apache Zookeeper hosts configured on cluster
- Three Apache Kafka hosts with one broker each
Each host is provisioned with JDK 7 and Kafka 0.8.1
Prerrequisites
- Vagrant
- VirtualBox
Setup
To start it up, just git clone this repo and execute vagrant up. Takes a while the first time as it will download all required dependencies for you.
Kafka is installed on /opt/kafka
Let's test it!
Login to any host with vagrant/vagrant. Some scripts have been included for convenience:
/vagrant/scripts/create_topic.sh <topic name>: this is a shortcut to create a topic using three replicas and one partition, executes/usr/local/kafka/bin/kafka-create-topic.sh --zookeeper 10.30.3.2:2181 --replica 3 --partition 1 --topic <topic name>- Topic details can be listed with
/usr/local/kafka/scripts/kafka-list-topic.sh --zookeeper 10.30.3.2:2181 /vagrant/scripts/producer.sh <topic name>: this will create a console producer, sending output to the topic created before, executes/usr/local/kafka/bin/kafka-console-producer.sh --broker-list 10.30.3.10:9092,10.30.3.20,10.30.3.30 --topic <topic name>/vagrant/scripts/consumer.sh <topic name>: this will create a console consumer, getting messages from the topic created before, executes/usr/local/kafka/bin/kafka-console-consumer.sh --from-beginning --zookeeper 10.30.3.2:2181 --topic <topic name>
Now if you input anything on the producer, it will show on the consumer.