Question on Spark 2.4.x support
groodt opened this issue · comments
Looking at the pom.xml and the release notes, it appears that version 0.2.1 supports Spark 2.4.x+ and Spark 3.x? With Scala 2.11. Is that correct?
This section of the README is misleading if so:
Version 0.1.x is based on Spark 2.3.
Version 0.2.x is based on Spark 3.0.
Given that Spark 3.0 also runs on Scala 2.12, is there likely to be a version released to bintray or maven central for 2.12? I realise that I can also build my own jar, I am just curious. Could we also not modify the pom.xml to publish for both 2.11 and 2.12?
I believe v0.2.1 supports Spark 2.4 as well, but we have not tested it thoroughly. Let me know if you see issues with Spark 2.4. I will change the README accordingly.
We can try to build a jar for Scala 2.12 if it does not involve a lot of code changes. But I don't know how to do this without changing the pom.xml file. Currently I am thinking to have a dedicated branch for Scala 2.12. If you know an easy way to do so, please let us know. Thanks.
Thanks @junshi15 We're not seeing any issues in our tests so far on 2.4.5, but I'll let you know after any further testing.
In terms of Scala cross-compilation, should we try something like this?
https://github.com/kelnos/scala-cross-maven-plugin
It seems to handle the variable interpolation and pom rewriting. Do you think it will work? Would you accept a PR that enables this?
Thanks for your suggestion. I have no experience with this plugin.
I am using IntelliJ, the limitation below is a bit annoying. But it is worth a try. Send in a PR whenever you are ready. I can test it out.
You do not put defaults for the above properties in the POM's main properties section. (Note: Some IDEs, like IntelliJ IDEA, won't work properly without a default for scala.binary.version, even if you properly select the correct profile to use. Adding a default for that property seems to work ok.)
I've created a PR here:
#12