Classpath issue : Could not find or load main class org.apache.spark.launcher.Main
majyphi opened this issue · comments
I'm trying to setup a spark-submit image to launch our jobs.
To test the setup I run & enter the latest spark-submit image with : docker run -it bde2020/spark-submit:latest bash
Once inside I try to start a spark-shell : /spark/bin/spark-shell
I am greeted with :
Error: Could not find or load main class org.apache.spark.launcher.Main
/spark/bin/spark-class: line 96: CMD: bad array subscript
Down the rabbit hole this is the line that fails in spark-class : java -Xmx128m -cp '/spark/jars/*' org.apache.spark.launcher.Main org.apache.spark.deploy.SparkSubmit --class org.apache.spark.repl.Main --name 'Spark shell'
It seems the -cp /spark/jars/* bit does not correctly incorporate the jars inside the directory.
Do you have any idea on the cause of this problem ?
EDIT : Running on Red Hat Enterprise Linux Server 7.8 (Maipo) this might be a factor as I have no issues on Ubuntu 20+
After investigation it was indeed caused by the OS, or the underlying system.