- generate-data service creates 100k rows in postgres
- kafka-producer service reads the data and sends it to kafka
- flink-processor runs at the same time with a tumbling window of 5 mins and processes the sum of the costs based on customer_name's
- if a tumbling window has a window interval of 10 seconds, every incoming event from a data stream for 10 seconds will fall into the same window
- then we compare the results from flink by running a sql query and it's accurate
.
- create jars in flink-processor folder
mvn clean install
- to start containers
docker-compose up -d
- kafka tricks:
kafka-topics.sh --bootstrap-server kafka:9092 --create --topic datastream# to check list of topics
kafka-topics.sh --bootstrap-server kafka:9092 --list- in code, for local testing, replace all the container name in url's with localhost. To make it work in docker, we need to make sure we have container names instead