apache / sedona

A cluster computing framework for processing large-scale geospatial data

Home Page:https://sedona.apache.org/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

1.5.0 spark shaded protobuf version conflict with application

hannin opened this issue · comments

Expected behavior

My application uses protobuf also with version higher than 3.19.x
with org.apache.sedona:sedona-spark-shaded-3.0_2.12:1.5.0, protobuf generated files makes error like

        if (!com.google.protobuf.GeneratedMessageV3.isStringEmpty(osVersion_)) {
                                                   ^
  symbol:   method isStringEmpty(java.lang.Object)
  location: class com.google.protobuf.GeneratedMessageV3

It's because protoc version and protobuf-java lib version not matched

Actual behavior

with org.apache.sedona:sedona-spark-shaded-3.0_2.12:1.5.0, custom protobuf-java version builds

Steps to reproduce the problem

I made sample project for reproduce https://oss.navercorp.com/tak/spark-sedona-test
./gradlew build failed there

Settings

Sedona version = 1.5.0

Apache Spark version = not explicit

Apache Flink version = no

API type = Scala

Scala version = 2.12

JRE version = 1.8

Python version = no

Environment = build phase


with shaded jar, there is relocate function to fix this (ref https://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html)

@hannin Since you are using gradle, you can use our unshaded jar: https://sedona.apache.org/1.5.0/setup/maven-coordinates/#use-sedona-unshaded-jars

Here is an example Scala project that uses the unshaded jar. You can exclude protobuf-java from the unshaded dependency. https://github.com/apache/sedona/blob/master/examples/spark-sql/pom.xml

oh there was documented solution already.. thank you