Powered by Rock the JVM!
Install and setup
- install IntelliJ IDEA
- install Docker Desktop
- either clone the repo or download as zip
- open with IntelliJ as an SBT project
As you open the project, the IDE will take care to download and apply the appropriate library dependencies.
To set up the dockerized Spark cluster we will be using in the course, do the following:
- open a terminal and navigate to
spark-cluster
- run
build-images.sh
(if you don't have a bash terminal, just open the file and run each line one by one) - run
docker-compose up
To interact with the Spark cluster, the folders data
and apps
inside the spark-cluster
folder are mounted onto the Docker containers under /opt/spark-data
and /opt/spark-apps
respectively.
To run a Spark shell, first run docker-compose up
inside the spark-cluster
directory, then in another terminal, do
docker exec -it spark-cluster_spark-master_1 bash
and then
/spark/bin/spark-shell