Settings for using Jupyter hub/notebook with CDH5 and PySpark
Including kernels for Spark inclued with CDH5 (currently 1.6) and the latest released version of Spark
- Python 3 &
pip3
pip3 install ipykernel
pip3 install py4j
- Java 8+ on all YARN workers
https://github.com/jupyterhub/jupyterhub
Kernels are located in /usr/local/share/jupyter/kernels
Move pysparkCDH
and pysparkLatest
folders into the above path
- Download the latest Spark version for Hadoop 2.6 (CDH5 is built on Hadoop 2.6):
- Extract tarball to
/opt/spark
- Upload Spark libs to HDFS for workers to pick up
hadoop fs -mkdir /lib
hadoop fs -put /opt/spark/jars/*.jar /lib
- Upload YARN Shuffle jar
hadoop fs -put /opt/spark/yarn /lib
Follow instructions here: http://spark.apache.org/docs/latest/running-on-yarn.html#configuring-the-external-shuffle-service
Add to yarn-site.xml
for yarn.application.classpath
- hdfs://hdfs-cluster:8020/lib/yarn/*
Configure JAVA_HOME
for all worker nodes to point to Java 8 install, this can be done in Cloudera Manager
Make sure the HDFS path is correct and adjust to your setup, then place into /opt/spark/conf