JAR version Issue while Implementing StreamingLense
rpatid10 opened this issue · comments
Rahul Patidar commented
I am trying to Implement StreamingLense In Spark Application.I Have added below 2 lines in existing code as suggested here. https://github.com/qubole/streaminglens
1. class StreamingLens_POC(spark: SparkSession, options: RequestBuilder){}
2. val streamingLens = new StreamingLens_POC(spark, options)
// Added New Block For StreamingLense
class StreamingLens_POC(spark: SparkSession, options: RequestBuilder)
// Existing Code which was working fine without any issue.
object StreamingLens_POC {
def main(args: Array[String]): Unit = {
val applicationName = args(0)
val spark = SparkSession
.builder()
.appName(applicationName)
//.config("spark.master", "local") //Addition code to execute in local
.getOrCreate()
println("Spark Streaming Lens POC Program Started")
val streamingLens = new StreamingLens_POC(spark, options) // added this new line for StreamingLense
//..... existing code Code....
..
..
..
..
}
After that When I am trying to execute this application on server using below spark submit Command.
spark-submit \
--name SPARK_STREAMING_POC \
--num-executors 1 \
--jars /home/username/jar/spark-streaminglens_2.11-0.5.3.jar , /home/username/jar/logstash-gelf-1.3.1.jar, ..(other required jar) \
--master yarn --deploy-mode cluster --driver-cores 1 --driver-memory 2G --executor-cores 1 --executor-memory 2G \
--supervise --class com.pkg.data.StreamingLens_POC /home/username/jar/PrjectJarName.jar \
SPARK_STREAMING_POC
But Its Giving Below Error.
21/09/24 11:50:26 ERROR ApplicationMaster: User class threw exception: java.lang.NoSuchMethodError: biz.paluch.logging.gelf.log4j.GelfLogAppender.setAdditionalFieldTypes(Ljava/lang/String;)V
java.lang.NoSuchMethodError: biz.paluch.logging.gelf.log4j.GelfLogAppender.setAdditionalFieldTypes(Ljava/lang/String;)V
Can someone Kindly Suggest. If I need to do any addition Task here.
Rahul Patidar commented
I have added biz.paluch.logging.gelf.log4j in spark submit summand using --jars option and it worked ,so closing the issue.
Thanks.