This Docker image can be used to run PySpark analyses with data stored in ADLS Gen2 Storage (Azure Data Lake Storage). This image can be run locally for development/testing or deployed to a large VM as a fast, simple, and low-cost method of running PySpark analyses on a single node.
See:
- Benchmarking Apache Spark on a Single Node Machine
- Run Spark Jobs on Azure Batch using Azure Container Registry and Blob storage
Before building the image, Python package dependencies are specified in the requirements.in
file and compiled using pip-tools to generate pinned package versions in requirements.txt
:
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install pip-tools
pip-compile requirements.in
Build the image:
docker build -t spark_adls .
Start the PySpark shell for interactive use:
docker run -it --rm \
-p 4040:4040 \
spark_adls \
/opt/spark/bin/pyspark
Run a Python script by attaching current directory as a volume and calling spark-submit:
docker run -it --rm \
-v $(pwd):/opt/spark/work-dir \
-p 4040:4040 \
spark_adls \
/opt/spark/bin/spark-submit --driver-memory 2g /opt/spark/work-dir/test.py