Powered by Rock the JVM!
- install IntelliJ IDEA
- install Docker Desktop
- either clone the repo or download as zip
- open with IntelliJ as an SBT project
As you open the project, the IDE will take care to download and apply the appropriate library dependencies.
To set up the dockerized Spark cluster we will be using in the course, do the following:
- open a terminal and navigate to
spark-cluster - run
build-images.sh(if you don't have a bash terminal, just open the file and run each line one by one) - run
docker-compose up
To interact with the Spark cluster, the folders data and apps inside the spark-cluster folder are mounted onto the Docker containers under /opt/spark-data and /opt/spark-apps respectively.
To run a Spark shell, first run docker-compose up inside the spark-cluster directory, then in another terminal, do
docker exec -it spark-cluster_spark-master_1 bash
and then
/spark/bin/spark-shell