This repo illustrates the Spark Structured Streaming
Gets random names from the API. Sends the name data to Kafka topics every 10 seconds using Airflow. Every message is read by Kafka consumer using Spark Structured Streaming and written to Cassandra table on a regular interval.
stream_to_kafka_dag.py
-> The DAG script that writes the API data to a Kafka producer every 10 seconds.
stream_to_kafka.py
-> The script that gets the data from API and sends it to Kafka topic
spark_streaming.py
-> The script that consumes the data fromo Kafka topic with Spark Structured Streaming
response.json
-> Sample response coming from the API
Run the following command to clone the necessary repo on your local
git clone https://github.com/dogukannulu/docker-airflow.git
After cloning the repo, run the following command only once:
docker build --rm --build-arg AIRFLOW_DEPS="datadog,dask" --build-arg PYTHON_DEPS="flask_oauthlib>=0.9" -t puckel/docker-airflow .
Then change the docker-compose-LocalExecutor.yml file with the one in this repo and add requirements.txt
file in the folder. This will bind the Airflow container with Kafka and Spark container and necessary modules will automatically be installed:
docker-compose -f docker-compose-LocalExecutor.yml up -d
Now you have a running Airflow container and you can access the UI at https://localhost:8080
docker-compose.yml
will create a multinode Kafka cluster. We can define the replication factor as 3 since there are 3 nodes (kafka1, kafka2, kafka3). We can also see the Kafka UI on localhost:8888
.
We should only run:
docker-compose up -d
![image](https://private-user-images.githubusercontent.com/91257958/248595143-2b55fbc9-d0f8-4919-880e-58c7f68c653d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ2MTQ2MDUsIm5iZiI6MTcyNDYxNDMwNSwicGF0aCI6Ii85MTI1Nzk1OC8yNDg1OTUxNDMtMmI1NWZiYzktZDBmOC00OTE5LTg4MGUtNThjN2Y2OGM2NTNkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODI1VDE5MzE0NVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTM0MTUzYWQ4MzYyNTdkOGM5YjRlNzg5YmJmMzQyNDNkNjQ1NTNkY2IzZDIwOTVjNjU5NjJjZmI0YjEwNDRlZDAmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.7RA8K46onR6CA9oCRTOxLOsDFDHudgZNEuvmHpnJX7c)
After accessing to Kafka UI, we can create the topic random_names
. Then, we can see the messages coming to Kafka topic:
![image](https://private-user-images.githubusercontent.com/91257958/248595134-37a4286a-2afe-49bd-845d-1c98d2b45e3f.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ2MTQ2MDUsIm5iZiI6MTcyNDYxNDMwNSwicGF0aCI6Ii85MTI1Nzk1OC8yNDg1OTUxMzQtMzdhNDI4NmEtMmFmZS00OWJkLTg0NWQtMWM5OGQyYjQ1ZTNmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODI1VDE5MzE0NVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWIyMzRjN2M0NGMyYzM0NWI2OGJiMzM5OTZmODRkNjY1NWM1MmIxZjcxMDg2ZDEzODM1NGUzYzcxZTEwNzJiNTImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.UREsENCKiEGszf4QjoFzopaESU9WDR5LEIa3i-QAnFg)
docker-compose.yml
will also create a Cassandra server. Every env variable is located in docker-compose.yml
. I also defined them in the scripts.
By running the following command, we can access to Cassandra server:
docker exec -it cassandra /bin/bash
After accessing the bash, we can run the following command to access to cqlsh cli.
cqlsh -u cassandra -p cassandra
Then, we can run the following commands to create the keyspace spark_streaming
and the table random_names
:
CREATE KEYSPACE spark_streaming WITH replication = {'class':'SimpleStrategy','replication_factor':1};
CREATE TABLE spark_streaming.random_names(full_name text primary key, gender text, location text, city text, country text, postcode int, latitude float, longitude float, email text);
DESCRIBE spark_streaming.random_names;
We should move stream_to_kafka.py
and stream_to_kafka_dag.py
scripts under dags
folder in docker-airflow
repo. Then we can see that random_people_names
appears in DAGS page.
When we turn the OFF button to ON, we can see that the data will be sent to Kafka topics every 10 seconds. We can check from Kafka UI as well.
![image](https://private-user-images.githubusercontent.com/91257958/248595196-8dc1ef5c-f256-40d3-94dc-7f6247edbe44.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ2MTQ2MDUsIm5iZiI6MTcyNDYxNDMwNSwicGF0aCI6Ii85MTI1Nzk1OC8yNDg1OTUxOTYtOGRjMWVmNWMtZjI1Ni00MGQzLTk0ZGMtN2Y2MjQ3ZWRiZTQ0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODI1VDE5MzE0NVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWE0MmRlMzJiYTEyYTlkMDhmYTdlNDRmZDYzNTM0ZjRjZTg2ZTBlYTYxZWJjM2FlODRkZDU5YTVjMmMyMDg3Y2MmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.lOAJpO__8cpjQkvR1nbNE4jzcZ3AcHKyj1_LXBpyju8)
First of all we should copy the local PySpark script into the container:
docker cp spark_streaming.py spark_master:/opt/bitnami/spark/
We should then access the Spark container and install necessary JAR files under jars directory.
docker exec -it spark_master /bin/bash
We should run the following commands to install the necessary JAR files under for Spark version 3.3.0:
cd jars
curl -O https://repo1.maven.org/maven2/com/datastax/spark/spark-cassandra-connector_2.12/3.3.0/spark-cassandra-connector_2.12-3.3.0.jar
curl -O https://repo1.maven.org/maven2/org/apache/spark/spark-sql-kafka-0-10_2.13/3.3.0/spark-sql-kafka-0-10_2.13-3.3.0.jar
While the API data is sent to the Kafka topic random_names
regularly, we can submit the PySpark application and write the topic data to Cassandra table:
cd ..
spark-submit --master local[2] --jars /opt/bitnami/spark/jars/spark-sql-kafka-0-10_2.13-3.3.0.jar,/opt/bitnami/spark/jars/spark-cassandra-connector_2.12-3.3.0.jar spark_streaming.py
After running the commmand, we can see that the data is populated into Cassandra table
Enjoy :)