felipeall / streaming-video-classifier

Kafka deployed locally in Kubernetes that simulates streaming videos, classifies the frames with ML and stores the predictions in MongoDB

Streaming Video Classifier

Project to demonstrate a Kafka deployment in a local Kubernetes environment that simulates streaming videos, classifies the frames images using a machine learning model and uploads the predictions to a MongoDB collection

Technologies

• Docker
• Minikube
• Kafka
• MongoDB
• OpenCV
• Tensorflow

Prerequisites

• Docker: open platform for developing, shipping, and running applications
• Minikube: local Kubernetes, focusing on making it easy to learn and develop

Setup

1. Clone the repository

   https://github.com/felipeall/streaming-video-classifier.git

2. Put videos files in the following folder:

   streaming-video-classifier/apps/video-producer/videos

   Supported formats are: avi, mp4 and webm

Running

tl;dr: make run

1. Start minikube service

   minikube start

2. Deploy Zookeeper and Kafka Broker

   kubectl apply -f kubernetes/zookeeper.yaml
   kubectl wait $(kubectl get pods -o name) --for=condition=Ready --timeout=600s
   kubectl apply -f kubernetes/kafka-broker.yaml

3. Deploy MongoDB

   kubectl apply -f kubernetes/mongodb.yaml

4. Build and deploy Video Consumer

   docker build -t video-consumer -f apps/video-consumer/Dockerfile apps/video-consumer
   minikube image load video-consumer
   kubectl apply -f kubernetes/video-consumer.yaml

5. Build and deploy Video Producer

   docker build -t video-producer -f apps/video-producer/Dockerfile apps/video-producer
   minikube image load video-producer
   kubectl apply -f kubernetes/video-producer.yaml

Validation

In order to check the output, enable MongoDB port forwarding:

make mongodb-local

And connect to the database locally via:

mongodb://localhost:27017