anandshivam44 / kafka-streaming-example

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Prerequisites

  • Java installed in host machines when running Kafka in Host machine
  • Docker installed on Host Machine
  • Java installed inside Docker container

Getting Started

git clone https://github.com/anandshivam44/kafka-streaming-example.git

For the sake of ease set work directory in all terminal session and use absolute paths to avoid errors

export WORK_DIR=$(pwd)/kafka-streaming-example

1. Start Services aka Start Apache Kafka in Host Machine

1.1 Start Zookeeper in new terminal

cd $WORK_DIR/kafka_2.13-2.8.0

bin/zookeeper-server-start.sh config/zookeeper.properties

1.2 Start Kafka Server in new terminal

cd $WORK_DIR/kafka_2.13-2.8.0

bin/kafka-server-start.sh config/server.properties

2. Run the streaming example:

2.1 Create the required Topics

NOTE: First time only, create the required topics.

cd $WORK_DIR/kafka-streams 

./bin/create-topics.sh $WORK_DIR/kafka_2.13-2.8.0 localhost 2181

2.2 Json Generator

  • Build the json generator container
cd $WORK_DIR/json-data-generator-1.4.1

docker build -t json-data-generator .
  • Run the JSON data generator docker container
docker run --network="host"  -it json-data-generator

3. Build and Run the streams which processes the orders.

3.1 Stream: PurchaseProcessor

  • Build the stream PurchaseProcessor
cd $WORK_DIR/kafka-streams

docker build -t stream-purchase-processor -f DockerfileRunPurchaseProcessor .
  • Run the docker container containing stream runPurchaseProcessor
docker run --network="host"  -it stream-purchase-processor

3.2 Stream: Purchase

  • Build the stream PurchaseStreams
cd $WORK_DIR/kafka-streams

docker build -t purchase-streams -f  DockerfileRunPurchaseStreams .
  • Run the docker container containing stream PurchaseStreams
docker run --network="host"  -it purchase-streams

4. Check the output topics

4.1 Topic: Purchases

  • Build the subscriber Purchases
cd $WORK_DIR/kafka_2.13-2.8.0 

docker build -t subscriber-purchases -f  DockerFileOutputTopicPurchases .
  • Run the docker container with Topic: Purchases
docker run --network="host"  -it subscriber-purchases

4.2 Topic: Rewards

  • Build the subscriber Rewards
cd $WORK_DIR/kafka_2.13-2.8.0 

docker build -t subscriber-rewards -f  DockerFileOutputTopicRewards .
  • Run the docker container with Topic: Rewards
docker run --network="host"  -it subscriber-rewards

4.3 Topic: Patterns

  • Build the subscriber Patterns
cd $WORK_DIR/kafka_2.13-2.8.0 

docker build -t subscriber-patterns -f  DockerFileOutputTopicPatterns .
  • Run the docker container containing stream PurchaseStreams
docker run --network="host"  -it subscriber-patterns

About


Languages

Language:Java 76.1%Language:Scala 8.1%Language:Roff 7.7%Language:Shell 4.5%Language:Batchfile 2.8%Language:Python 0.7%Language:Dockerfile 0.0%