Algebird-spark with Spark 2.{3, 4} and Scala 2.12
ktrivedisf opened this issue · comments
We are trying to compile algebird-spark with Spark 2.4.0 and Scala 2.12. We are seeing following errors
error] Error while emitting AlgebirdRDD.scala [error] 15859 [error] Error while emitting AlgebirdRDD.scala [error] 10832 [error] Error while emitting package.scala [error] assertion failed: ClassBType.info not yet assigned: Lorg/apache/spark/rdd/RDD; [error] Error while emitting package.scala [error] assertion failed: ClassBType.info not yet assigned: Lorg/apache/spark/rdd/RDD; [error] Error while emitting package.scala [error] assertion failed: ClassBType.info not yet assigned: Lorg/apache/spark/rdd/RDD;
We do have following changes in build.sbt
- revert 0569d2b#diff-fdc3abdfd754eeb24090dbd90aeec2ce (mainly uncomment algebirdSpark on line 187)
- Remove https://github.com/twitter/algebird/blob/develop/build.sbt#L49 (reference of 2.10)
- Remove https://github.com/twitter/algebird/blob/develop/build.sbt#L246 (filter of 2.12)
- Change https://github.com/twitter/algebird/blob/develop/build.sbt#L245 from Spark 1.3 to 2.4
Spark 2.4.0 got released (RC5) with Scala 2.12 So I was wondering if there are any plans on that front to make algebird-spark compile with Spark 2.4 or any general direction you would recommend ?
I’d welcome a PR but we just haven’t gotten to it yet.
Shouldn’t be too much work.
As of 0.13.5 algebird-spark is cross-compiled to 2.12 and spark >=2.4.0