HeartSaVioR / spark-sql-kafka-offset-committer

Kafka offset committer for structured streaming query

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

ERROR Utils: uncaught error in thread spark-listener-group-streams

kishoreraj05 opened this issue · comments

Getting exception while using spark-sql-kafka-offset-committer
Version:

            <groupId>net.heartsavior.spark</groupId>
            <artifactId>spark-sql-kafka-offset-committer_2.12</artifactId>
            <version>0.4.0-spark-3.0</version>
        </dependency>

Error:
23/12/06 05:25:45 ERROR Utils: uncaught error in thread spark-listener-group-streams, stopping SparkContext
java.lang.NoSuchMethodError: 'org.apache.spark.sql.connector.read.InputPartition org.apache.spark.sql.execution.datasources.v2.DataSourceRDDPartition.inputPartition()'
at org.apache.spark.sql.kafka010.KafkaSourceInspector.$anonfun$extractSourceTopicsFromDataSourceV2$2(KafkaSourceInspector.scala:112)
at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at scala.collection.TraversableLike.flatMap(TraversableLike.scala:293)
at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:290)
at scala.collection.mutable.ArrayOps$ofRef.flatMap(ArrayOps.scala:198)
at org.apache.spark.sql.kafka010.KafkaSourceInspector.$anonfun$extractSourceTopicsFromDataSourceV2$1(KafkaSourceInspector.scala:111)
at org.apache.spark.sql.kafka010.KafkaSourceInspector.$anonfun$extractSourceTopicsFromDataSourceV2$1$adapted(KafkaSourceInspector.scala:110)
at scala.collection.immutable.List.flatMap(List.scala:366)
at org.apache.spark.sql.kafka010.KafkaSourceInspector.extractSourceTopicsFromDataSourceV2(KafkaSourceInspector.scala:110)
at org.apache.spark.sql.kafka010.KafkaSourceInspector.$anonfun$populateKafkaParams$1(KafkaSourceInspector.scala:40)
at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)