BParesh89 / spark-playground

Template project for spark-scala in intellij with maven.

This project can be used as a template to create a scala-spark project in intellij using maven for build. Also, this project has sample code, docker files and scripts to run spark master , worker nodes and history server on docker.

After building the project, jar would go to target folder.

Contents

1. Spark Join Demo

2. Read Json

3. Read Nested Json - explode demo

4. Read & write data in AWS S3 bucket

Reference

1. http://knowdimension.com/en/data/create-a-spark-application-with-scala-using-maven-on-intellij/

2. https://www.pavanpkulkarni.com/blog/13-spark-on-docker/