AndyGuoxingWANG / Python-and-Spark-for-Big-Data-master

Python-and-Spark-for-Big-Data

Course Notebooks for Python and Spark for Big Data

Course Outline:

• Course Introduction

  • Promo/Intro Video
  • Course Curriculum Overview
  • Introduction to Spark, RDDs, and Spark 2.0

• Course Set-up

  • Set-up Overview
  • EC2 Installation Guide
  • Local Installation Guide with VirtualBox
  • Databricks Notebooks
  • Unix Command Line Basics and Jupyter Notebook Overview

• Spark DataFrames

  • Spark DataFrames Section Introduction
  • Spark DataFrame Basics
  • Spark DataFrame Operations
  • Groupby and Aggregate Functions
  • Missing Data
  • Dates and Timestamps

• Spark DataFrame Project

  • DataFrame Project Exercise
  • DataFrame Project Exercise Solutions

• Machine Learning

  • Introduction to Machine Learning and ISLR
  • Machine Learning with Spark and Python and MLlib
  • Consulting Project Approach Overview

• Linear Regression

  • Introduction to Linear Regression
  • Discussion on Data Transformations
  • Linear Regression with PySpark Example (Car Data)
  • Linear Regression Consulting Project (Housing Data)
  • Linear Regression Consulting Project Solution

• Logistic Regression

  • Introduction to Logisitic Regression
  • Logistic Regression Example
  • Logistic Regression Consulting Project (Customer Churn)
  • Logistic Regression Consluting Project Solution

• Tree Methods

  • Introduction to Tree Methods
  • Decision Tree and Random Forest Example
  • Random Forest Classification Consulting Project - Dog Food Data
  • RF Classification Consulting Project Solutions
  • RF Regression Project - (Facebook Data)

• Clustering

  • Introduction to K-means Clustering
  • Clustering Example - Iris Dataset
  • Clustering Consulting Project - Customer Segmentation (Fake Data)
  • Clustering Consulting Project Solutions

• Recommender System

  • Introduction to Recommender Systems and Collaborative Filtering
  • Code Along Project - MovieLens Dataset
  • Possible Consulting Project ? Company Service Reviews

• Natural Language Processing

  • Introduction to Project/NLP/Naive Bayes Model
  • What are pipelines?
  • Code Along

• Spark Streaming

  • Introduction to Spark Streaming
  • Spark Streaming Code-along!