Bigkube is about big data and Spark local development automation - automated deployments, well fitted components, integration testing with SBT right from IDE console.
- Install Docker
- Install Minikube
- Get Helm
- Get Scala and SBT
- Make sure SBT version is not less than 1.2.8 and there's Scala 2.11 sdk is set for the project
- Make sure minikube has --cpus=4 --memory=8192 flags set. Make sure the appropriate vm-driver is set (use
minikube config set). - Run
sbt assmblyrepo's base dir.
cd deployment- DON'T SKIP THIS./bigkube.sh --serve-jar- read output instructions carefully, ensure jar serving host ip is substituted according instructions./bigkube.sh --create- creates all necessary infrastructure. Troubleshooting: don't worry if some pods are "red" right after deployment. All Hadoop deployments are smart enough to wait for each other to be stand by in appropriate sequence. Just give them some time../bigkube.sh --spark-init- inits Helm (with Tiller) and Spark operator.
Note: bigkube.sh resolves all service accounts issues, secrets, config maps and etc.
- Write your own integration test using
SparkController. Examples provided. - Simply run
sbt it:testfrom repo's base dir - and that's it, your Spark app is deployed into minikube and tests are executed locally on your host machine.
minikube dashboard- Run
minikube services list. You can go tonamenodeandprestoUI with corresponding URLs. - One can use Metabase, an open source tool for rapid access data. Works with Presto and SQLServer as well.
Alongside with kubectl delete -f file.yaml you can use:
./bigkube.sh --delete- deletes all bigkube infrastructure./bigkube.sh --spark-drop- deletes helmed spark operator
Thanks to Nick Grigoriev for the idea and help. Thanks to Big Data Europe for Hadoop Docker images. Thanks to Valeira Katalnikova for Bigkube logo.