ERROR Build and Tests failing for Spark 3.5
galakfryar opened this issue · comments
Hi there HeartSaVioR.
You are a legend in this space for providing a library to commit offsets back to Kafka.
I have various businesses that require monitoring of consumers and this is a lifesaver.
Recently I looked into upgrading this library to Spark 3.5.
Through my investigation, I found that they've been significant changes in Spark that now break this library for potential upgrade.
Specifically here: https://github.com/apache/spark/blob/v3.5.0/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceRDD.scala#L32
inputPartition is now inputPartitions which is a Seq of Partitions :(
There are definitely a lot more changes that break the tests as it stands. So will be stuck at Spark 3.0.x.
I've already tried the other versions up to 3.5.x
I'll be reading this code more thoroughly as time allows and seeing if I have the opportunity to make changes and keep this alive.
Otherwise, just letting you know and to also make this visible to you and others.