dilo00o / data-engineering

Spark, Airflow, Kafka, Data engineering Nanodegree, Data Streaming Nanodegree

Data Engineering

Jun Zhu

Dev Environment Setup

• Instruction of setting up a standalone Spark cluster locally in Docker.

• Instruction of running Airflow in Docker.

• Instruction of running Kafka in Docker.

Udacity Data Engineering Nanodegree

• Data Modeling with postgreSQL

• Data Modeling with Apache Cassandra

• Data Warehouse with AWS Redshift

  

• Data Lake with Apache Spark

  

• Data Pipeline with Apache Airflow

  

• Data Engineering Capstone Project

  

Udacity Data Streaming Nanodegree

• Transit Status Tracker with Kafka

  

• STEDI app with Spark Streaming