This repository contains project related to Udacity's Data Engineering Nanodegree program. It consists of the projects completed as a part of the Nanodegree Program.
-
Data Modeling with Postgres: Modeling user activity data for a music streaming app called Sparkify.Creating a relational database and ETL pipeline designed to optimize queries for understanding what songs users are listening to. You’ll model your data in Apache Cassandra to allow for specific queries provided by the analytics team at Sparkify.
-
Data Modeling with Apache Cassandra: Creating a noSQL database and ETL pipeline designed to optimize queries for understanding what songs users are listening to and modeling the data in Apache Cassandra to allow for specific queries provided by the analytics team at Sparkify.
-
Data Warehousing with AWS: Building an ETL pipeline that extracts data from S3, stages them in Redshift, and transforms data into a set of dimensional tables for their analytics team to continue finding insights in what songs their users are listening to.
-
Data Lake with Spark: Implementing an ETL pipeline for a data lake. The data resides in S3, in a directory of JSON logs on user activity on the app, as well as a directory with JSON metadata on the songs in the app. You will load data from S3, process the data into analytics tables using Spark, and load them back into S3. You'll deploy this Spark process on a cluster using AWS.
-
Data Pipelines with Airflow: Creating and automating a set of data pipelines. You’ll configure and schedule data pipelines with Airflow and monitor and debug production pipelines.