sounishnath003 / usekafka

In this, I build a node-ts app which create topics, produce messages, consume. Which will help anybuddy to kick start and learn kafka including myself. Concepts collected from various sources and tutorials on internet.

UseKafka

Kafka created to gain and learn the concepts of kafka how it helps to build bus using realtime / stream-processing. Kafka made to Scala and Java. Original Author LinkedIn, later OpenSourced.

In this demo, I build a node-ts app which create topics, produce messages, consume. Which will help anybuddy to kick start and learn kafka including myself. Concepts collected from various sources and tutorials on internet.

To Know More: Wikipedia

Components

• Kafka Broker
• Kafka Producer
• Kafka Consumer
• Kafka Partitions
• Queue vs Pub-Sub
• Consumer Group
• ZooKeeper
• More in Documenentations

Pros

Apache Kafka is a community distributed event streaming platform capable of handling trillions of events a day. Initially conceived as a messaging queue, Kafka is based on an abstraction of a distributed commit log. Since being created and open sourced by LinkedIn in 2011, Kafka has quickly evolved from messaging queue to a full-fledged event streaming platform.

Aims to build unified, high-throughput, low-latency platform for handling real-time data feeds.

• Append only Commit Logs
• Performance
• Distributed
• Event Driven . Queues . Publish-Subcriber Pattern
• Long Poling
• Scaling
• Parallel Processing

Connections with DataSources

Cons

• Zookeeper
• Producer Explicit Partition can lead to problem in big processings
• Complex to install, configure, manage.

Running in Local

To run this project, You need to have Docker installed in your machine. I used docker to avoid the heavy installation of these stuffs.

• Installs ZooKeeper and Kafka from DockerHub

./setup.sh

• Install Dependencies

yarn

• Devphase Runner

yarn dev

• Build

yarn build

hostname: hostname of your machine

Acknowledgements

• Kafka Docs
• KafkaJs Npm Module
• Hussien Nassar