ffinfo / spark-sge

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

spark-sge

This script will launch a spark cluster on a sge cluster.

Arg Destription Default
-d Log directory $PWD/.spark
-e Number of executors 4
-c Number of CPU's per executor 4
-m Memory per executor in GB 20
-l Resource tag h_vmem
-p Parallel environment (required) -

On ctrl-c the executor jobs will be killed.

Example:

# This will start a cluster with 20 executors with 4 cpu's each, 80 cpu's in total and 400Gb of memory
spark-start-sge-cluster -e 20 -p <pe>
# This will start a cluster with 5 executors with 5 cpu's each, 25 cpu's in total and 200Gb of memory
spark-start-sge-cluster -e 5 -m 40 -c 5 -p <pe>

About

License:MIT License


Languages

Language:Shell 100.0%