chufnagel / flask_kafka

Kafka and Python

Working through the Build a Distributed Streaming System with Apache Kafka and Python tutorial by Mwaleh Muturi

A brief experiment with kafka-python. A short video is converted to a series of .png files, sent through Kafka, and visualized on a Flask server.

Note: Currently encountering issues with FFMPEG on Linux distros; can by bypassed via Docker or compiling opencv with FFMPEG.

Requirements

• Pipenv

• Python 3.4 - 3.6

• Kafka

• OpenCV with FFMPEG (Linux distros only)

Dependencies

See Pipfile for more information.

• kafka-python
• opencv-python
• Flask
• flake8

Usage Instructions

1. pipenv install
2. Start local Kafka cluster in one terminal tab:

zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties & kafka-server-start /usr/local/etc/kafka/server.properties

1. Run producer.py in another terminal tab
2. Run consumer.py in another terminal tab
3. Navigate to http://localhost:5000 to see the corgis run!

The kafka client address can be modified in config.yml.

Acknowledgements

• Video shamelessly cribbed from Anne Indergaard