Not Able to See StreamingLens Report In Logs.
rpatid10 opened this issue · comments
Hi,
I am trying to Implement StreamingLens In My existing Streaming Application. My application Is working fine and its loading data from one kafka topic to anothe kafka topic. But in ambari I am not able to see StreamingLens Reports, when I did this for batch application using sparklence i could see the logs generated by sparklense with all the resources information but same report I am not able to see for streaming application can someone suggest if I need to do additional code or where should I check the reports which should generate by StreamingLens.
My Sample code
class SparkStreamingLens(spark: SparkSession, options: RequestBuilder)
object SparkStreamingLens {
def main(args: Array[String]): Unit = {
println(" Spark Parameters are :")
val igniteDetails = args(0)
val applicationName = args(1)
val argumentTable = args(2)
// options.addParameter("streamingLens.reporter.intervalMinutes", "1")
val spark = SparkSession
.builder()
.appName(applicationName)
.getOrCreate()
val streamingLens = new SparkStreamingLens(spark, options)
// Remaining Code to Read from Kafka and write Into Kafka(Streaming Data)
}
}
spark-submit Command:
spark-submit \
--verbose \
--name SparkStreamingLens \
--num-executors 1 \
--conf streamingLens.reporter.intervalMinutes=1 \
--jars /home/abc/jars/spark-streaminglens_2.11-0.5.3.jar,\
/home/abc/jars/kafka-clients-0.10.2.1.jar,\
--master yarn \
--deploy-mode cluster \
--driver-cores 1 --driver-memory 2G --executor-cores 1 --executor-memory 2G \
--supervise --class com.data.datalake.SparkStreamingLens \
/home/abc/jar/SparkStreamingLens-spark-utility_2.11-1.0.jar \
"jdbc:ignite:thin://00.000.00.00:00000;distributedJoins=true;user=aaaaaa;password=aaaaaaa;" \
SparkStreamingLens \
argumentTable