This project demonstrates how to set up and run a local Kafka server and use it alongside a Flask application to handle streaming of latitude and longitude data in real-time. The project involves producing, consuming, and visualizing geospatial data using Kafka and Leaflet.js.
Before you begin, ensure you have the following installed on your system:
- Homebrew (for macOS users)
- Java (OpenJDK 11)
- Apache Kafka
- Python 3.x and
venv(Python virtual environment)
-
Install Homebrew (if it's not already installed):
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" -
Install OpenJDK 11 using Homebrew:
brew install openjdk@11
-
Download Kafka from the official Kafka website.
-
Extract the downloaded files to a directory of your choice.
accessToken: MAP_ACCESS_TOKEN (in leaf.js file)
Access the token by signin to https://account.mapbox.com/
In this project Kafka relies on Zookeeper to manage distributed configurations. Follow these steps to start Zookeeper and Kafka:
-
Start Zookeeper (in one terminal):
bin/zookeeper-server-start.sh config/zookeeper.properties
-
Start Kafka Server (in another terminal):
bin/kafka-server-start.sh config/server.properties
-
Create a Python Virtual Environment (optional but recommended):
python3 -m venv venv source venv/bin/activate -
Install the necessary Python packages:
pip install -r requirements.txt
-
Run the Flask app:
python app.py
To create a new Kafka topic for your geospatial data:
bin/kafka-topics.sh --create --topic geodata_final123 --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1Run First Producer (in one terminal):
python busdata1.py
Run Second Producer (in second terminal):
python busdata2.py
Run First Producer (in third terminal):
python busdata3.py