Sayantanmukherjee6 / KEK

A docker compose file containing Kafka Elasticsearch and Kibana along with a Confluent Kafka connector.

KEK

• To start Kafka-ElasticSearch-Kibana with Kafka-Connector run:
  • docker-compose -f docker-compose-KEK.yaml up -d

Example: Logging system using KEK

• Start Fluentd agent (td-agent.conf provided in this repo). Other log collection framework can also be used.

• Start docker-compose-KEK.yaml.

• Change /etc/hosts:

  • 

• Create the Kafka-Connector by using Kafka-Connect API (A sample JSON is provided : connector.json).

• Dump system/code logs in /tmp/logs/ directory.

• Open Kibana (http://localhost:5601) and configure index pattern for the current topic. After it is configured properly, in the Discover tab of Kibana the logs will be displayed.

Installing and Configuring Fluentd

• https://docs.fluentd.org/installation/install-by-deb
• td-agent.conf is located at /etc/td-agent/td-agent.conf
• After modification of td-agent.conf file run sudo service td-agent restart.

Sources

• https://medium.com/@raymasson/kafka-elasticsearch-connector-fa92a8e3b0bc
• https://www.cloudboxlabs.com/blog/distributed-log-analytics-using-apache-kafka-kafka-connect-and-fluentd
• https://blog.treasuredata.com/blog/2015/07/07/collecting-docker-logs-with-fluentd/