spark-sge
This script will launch a spark cluster on a sge cluster.
Arg | Destription | Default |
---|---|---|
-d | Log directory | $PWD/.spark |
-e | Number of executors | 4 |
-c | Number of CPU's per executor | 4 |
-m | Memory per executor in GB | 20 |
-l | Resource tag | h_vmem |
-p | Parallel environment (required) | - |
On ctrl-c the executor jobs will be killed.
Example:
# This will start a cluster with 20 executors with 4 cpu's each, 80 cpu's in total and 400Gb of memory
spark-start-sge-cluster -e 20 -p <pe>
# This will start a cluster with 5 executors with 5 cpu's each, 25 cpu's in total and 200Gb of memory
spark-start-sge-cluster -e 5 -m 40 -c 5 -p <pe>