error in running job:"Exception in thread "main" java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$class"

Question

error in running job:"Exception in thread "main" java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$class"

wanghy73 opened this issue 6 years ago · comments

##after submit spark job, I got following error:
18/05/18 15:18:00 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$class
at com.linkedin.photon.ml.index.DefaultIndexMap.(DefaultIndexMap.scala:27)
at com.linkedin.photon.ml.index.DefaultIndexMapLoader.(DefaultIndexMapLoader.scala:26)
at com.linkedin.photon.ml.io.deprecated.GLMSuite.createDefaultIndexMapLoader(GLMSuite.scala:183)
at com.linkedin.photon.ml.io.deprecated.GLMSuite.readLabeledPointsFromAvro(GLMSuite.scala:120)
at com.linkedin.photon.ml.io.deprecated.AvroInputDataFormat.loadLabeledPoints(AvroInputDataFormat.scala:41)
at com.linkedin.photon.ml.Driver.prepareTrainingData(Driver.scala:233)
at com.linkedin.photon.ml.Driver.preprocess(Driver.scala:310)
at com.linkedin.photon.ml.Driver.run(Driver.scala:170)
at com.linkedin.photon.ml.Driver$.main(Driver.scala:699)
at com.linkedin.photon.ml.Driver.main(Driver.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: scala.collection.GenTraversableOnce$class
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

Alex Shelkovnykov · Answer 1 · Wed May 30 2018 05:38:35 GMT+0800 (China Standard Time)

Can you please double-check the version of Scala specified in the settings.gradle? LinkedIn uses custom Spark jars which downgrade to Scala 2.10 - however Spark 2.1.0 actually depends on Scala 2.11.

wanghy73 · Answer 2 · Thu May 31 2018 04:22:52 GMT+0800 (China Standard Time)

Alex, Thank you. I downgraded to 2.10 for Scala in spark and it is working fine. Regards haiyan

…

On Tue, May 29, 2018 at 2:38 PM, Alex Shelkovnykov ***@***.*** > wrote: Can you please double-check the version of Scala specified in the settings.gradle? LinkedIn uses custom Spark jars which downgrade to Scala 2.10 - however Spark 2.1.0 actually depends on Scala 2.11. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#363 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AlQlMaADn2QJdny4bOk56LOhEsb-X8OWks5t3b_egaJpZM4UF8jS> .

Alex Shelkovnykov · Answer 3 · Fri Jun 01 2018 06:26:07 GMT+0800 (China Standard Time)

I actually meant that you should upgrade the Scala version in the Gradle settings to 2.11.8, but if it's working it's working!