vicchugu / spark-kubernetes

spark on kubernetes

Deploying Spark on Kubernetes

Want to learn how to build this?

Check out the post.

Want to use this project?

Minikube Setup

Install and run Minikube:

1. Install a Hypervisor (like VirtualBox or HyperKit) to manage virtual machines
2. Install and Set Up kubectl to deploy and manage apps on Kubernetes
3. Install Minikube

Start the cluster:

$ minikube start --memory 8192 --cpus 4
$ minikube dashboard

Build the Docker image:

$ eval $(minikube docker-env)
$ docker build -t spark-hadoop:2.2.1 -f ./docker/Dockerfile ./docker

Create the deployments and services:

$ kubectl create -f ./kubernetes/spark-master-deployment.yaml
$ kubectl create -f ./kubernetes/spark-master-service.yaml
$ kubectl create -f ./kubernetes/spark-worker-deployment.yaml
$ minikube addons enable ingress
$ kubectl apply -f ./kubernetes/minikube-ingress.yaml

Add an entry to /etc/hosts:

$ echo "$(minikube ip) spark-kubernetes" | sudo tee -a /etc/hosts

Test it out in the browser at http://spark-kubernetes/.