Support Spark 3.0.0
jamesprinc3 opened this issue · comments
Hello,
With Spark 3.0.0 released, it would be great if this library could be updated to be compatible. I have made some progress in this PR (#3), however there is one question which comes to mind:
- How can Spark 2.x / Scala 2.11 be supported alongside Spark 3.0.0 / Scala 2.12. I think some extra complexity is needed in the build script (e.g. taking inspiration from scalatest: https://github.com/scalatest/scalatest/blob/3.2.x-new/build.xml#L82) to create artifacts for both?
Thanks for your PR. Building two versions with the same code will be nice, but it is quite complex from the link you shared. I won't have time to work on this in the near future. Contributions are welcome.
The master branch is for Spark-3.0.0, corresponding to published artifact version 0.2.x.
https://search.maven.org/artifact/com.linkedin.sparktfrecord/spark-tfrecord_2.11/0.2.1/jar