Implementation Of StreamingLens in Existing Spark Streaming Applications

[Rahul Patidar]

I am trying to Implement StreamingLense In Spark Application.I Have added below 2 lines in existing code as suggested here. https://github.com/qubole/streaminglens

    1. class StreamingLens_POC(spark: SparkSession, options: RequestBuilder){}
    2. val streamingLens = new StreamingLens_POC(spark, options) 

    // Added New Block For StreamingLense
    class StreamingLens_POC(spark: SparkSession, options: RequestBuilder)

   // Existing Code which was working fine without any issue.
    object StreamingLens_POC {
    def main(args: Array[String]): Unit = {
    val applicationName = args(0) 
    val spark = SparkSession
   .builder()
   .appName(applicationName)
   //.config("spark.master", "local") //Addition code to execute in local
   .getOrCreate()
  println("Spark Streaming Lens POC Program Started")
  val streamingLens = new StreamingLens_POC(spark, options)   // added this new line for StreamingLense
 //..... existing code Code....
..
..
..
..
}

After that When I am trying to execute this application on server using below spark submit Command.

    spark-submit \
    --name SPARK_STREAMING_POC \
    --num-executors 1 \
    --jars  /home/username/jar/spark-streaminglens_2.11-0.5.3.jar , /home/username/jar/logstash-gelf-1.3.1.jar, ..(other required jar) \
    --master yarn --deploy-mode cluster --driver-cores 1 --driver-memory 2G --executor-cores 1 --executor-memory 2G \
    --supervise --class com.pkg.data.StreamingLens_POC /home/username/jar/PrjectJarName.jar \
    SPARK_STREAMING_POC

But Its Giving Below Error.

     21/09/24 11:50:26 ERROR ApplicationMaster: User class threw exception: java.lang.NoSuchMethodError: biz.paluch.logging.gelf.log4j.GelfLogAppender.setAdditionalFieldTypes(Ljava/lang/String;)V
    java.lang.NoSuchMethodError: biz.paluch.logging.gelf.log4j.GelfLogAppender.setAdditionalFieldTypes(Ljava/lang/String;)V

Can someone Kindly Suggest. If I need to do any addition Task here.