yuzutech / kafka-connect-elasticsearch

Stream data from Apache Kafka to Elasticsearch using Kafka Connect

Kafka Connect Elasticsearch

Stream data from Apache Kafka to Elasticsearch using Kafka Connect.

Usage

1. Install Elasticsearch 2.3.3

   cd elasticsearch-2.3.3
   bin/elasticsearch

2. Install Confluent Platform 2.0.1

   cd confluent-2.0.1

   terminal 1

   /bin/zookeeper-server-start ./etc/kafka/zookeeper.properties

   terminal 2

   /bin/kafka-server-start ./etc/kafka/server.properties

3. Clone this project and build

   git clone https://github.com/yuzutech/kafka-connect-elasticsearch.git
   cd kafka-connect-elasticsearch
   mvn clean package

4. Export CLASSPATH and launch Kafka connect in standalone mode

   export CLASSPATH=/path/to/kafka-connect-elasticsearch/workspace/target/kafka-connect-elasticsearch-all.jar
   cd confluent-2.0.1
   cp /path/to/kafka-connect-elasticsearch/workspace/samples/config/standalone.properties .
   cp /path/to/kafka-connect-elasticsearch/workspace/samples/config/connector.properties .
   bin/connect-standalone standalone.properties connector.properties

5. Start Kafka REST Proxy

   terminal 3

   bin/kafka-rest-start

6. Produce JSON messages

   curl -X POST -H "Content-Type: application/vnd.kafka.json.v1+json" \
         --data '{"records":[{"value":{"foo":"bar"}}]}' "http://localhost:8082/topics/baz"

7. View your data in Elasticsearch!

   curl -XGET http://localhost:9200/qux/_search