7125messi / learning-apache-spark

This repository contains apache spark tutorials implemented with pypsark. For some machine learning methods, there will be comparisons between pyspark and R results.

Learning apache spark

Ming Chen & Wenqiang Feng

Introduction

This repository contains mainly notes from learning Apache Spark by Ming Chen & Wenqiang Feng. We try to use the detailed demo code and examples to show how to use pyspark for big data mining. If you find your work wasn't cited in this note, please feel free to let us know.

Content

• Cheat Sheets

  • Python Basics
  • Pandas Basics
  • Numpy Basics
  • Scikit-Learn
  • RDD Basics
  • DataFrame Basics
  • Apache Spark Cheat Sheet

• Data Manipulation

  • Entry Points to Spark
  • RDD Object
  • DataFrame Object
  • RDD and DataFrame conversion
  • Categorical Data, StringIndexer and OneHotEncoder
  • Continuous variables to categorical variables
  • Import and export data
  • Subset data:
    • select rows by index
    • select rows by logical criteria
    • select columns by index
    • select columns by names
    • select columns by regex pattern
  • udf() function and SQL data types:
    • use udf() function
    • difference between ArrayType and StructType
  • Pipeline
  • Dense and sparse vectors
  • Assemble feature columns into a featuresCol column with VectorAssembler
  • TF-IDF, HashingTF and CountVectorizer
  • Feature processing:
    • First data check
  • SQL functions
  • Add py Files to cluster

• Machine Learning

  • Machine Learning Framework

  • Regression

    • Linear regression
    • Logistic regression

  • Classification

    • Naive bayes classification
    • Decision tree
    • Random forest classification
    • Gradient boost tree classification

• Model Tuning

  • Regularization
  • Cross-validation

• Nutural Language Processing

  • NLP and NLTK Basics
  • NLP Information Extraction

Acknowledgement

At here, we would like to thank Jian Sun and Zhongbo Li at the University of Tennessee at Knoxville for the valuable disscussion and thank the generous anonymous authors for providing the detailed solutions and source code on the internet. Without those help, this repository would not have been possible to be made. Wenqiang also would like to thank the Institute for Mathematics and Its Applications (IMA) at University of Minnesota, Twin Cities for support during his IMA Data Scientist Fellow visit.

Feedback and suggestions

Your comments and suggestions are highly appreciated. We are more than happy to receive corrections, suggestions or feedbacks through email (Ming Chen: [e-mail], Wenqiang Feng: [e-mail]) for improvements.