| Authors | Date | Title |
|---|---|---|
Abhijit C Patil, Abhilasha Pandey, Ankit Agarwal, Ao Liu, Suzanne Kaminski, |
December 06, 2017 |
Helping Sun Country understand its customers using clustering and LDA on PySpark AND A tutorial to demonstrate Random Forest using PySpark |
Introduction
Data analytics and machine learning are one of the most important applications of distributed computing. spark.ml is a new package introduced in Spark 1.2+, which aims to provide a uniform set of high-level APIs that help users create and tune practical machine learning pipelines.We wanted to learn and explore SparkML in detail. To better demonstrate the vast capabilities in spark.ml, three tutorials were created that cover both supervised and unsupervised techniques. Algorithms included are Random Forest, Bisecting k-means and LDA. Our project contained topics within both Categories A and B.
To demonstrate the power of Spark and AWS together and show how they leverage big data sets, the project completed customer segmentation of a Sun Country Airlines dataset. In concert with analyzing the customer segments and to provide a more complete business solution to Sun Country, topic modeling and sentiment analysis were conducted after scraping airline reviews from the web.
Project Report
You can go through the following to access the tutorials