This is a project for Udacity Data Streaming Nanaodegree. It processes Chicago Transit Authority data providing a dashboard displaying train arrivals in real or near-real time. The goal is to consume the data using Apache Kafka, Faust and Python The dataset is publicly available via the Chicago Transit Authority.
Here's a screenshot of the expected dashboard:
The development requires the libraries and tool along with their dependencies:
- Python 3.7
- Apache Kafka
- Postgresql 11
- Faust
- Python libraries as specified in requirements.txt in the Producers and Consumers directories.
- Access to a computer with a minimum of 16gb+ RAM and a 4-core CPU to execute the simulation
Using the
The Chicago Transit Authority (CTA) has asked us to develop a dashboard displaying system status for its commuters. We have decided to use Kafka and ecosystem tools like REST Proxy and Kafka Connect to accomplish this task.
Our architecture will look like so:
python producers/simulation.py
cd consumers
faust -A faust_stream worker -l info
python consumers/ksql.py
python consumers/server.py