CamilleMo / Gitpod-spark-training

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Powered by Rock the JVM!

Install and setup

As you open the project, the IDE will take care to download and apply the appropriate library dependencies.

To set up the dockerized Spark cluster we will be using in the course, do the following:

  • open a terminal and navigate to spark-cluster
  • run build-images.sh (if you don't have a bash terminal, just open the file and run each line one by one)
  • run docker-compose up

To interact with the Spark cluster, the folders data and apps inside the spark-cluster folder are mounted onto the Docker containers under /opt/spark-data and /opt/spark-apps respectively.

To run a Spark shell, first run docker-compose up inside the spark-cluster directory, then in another terminal, do

docker exec -it spark-cluster_spark-master_1 bash

and then

/spark/bin/spark-shell

About


Languages

Language:Scala 58.6%Language:Dockerfile 27.8%Language:Shell 13.6%