support in spark 3.x version
abdul2011 opened this issue · comments
Does sparklens supported with spark 3.x version
i need to know this as well.
i tried running a simple WordCount application with spark-3.2.1 and kubernetes-1.20 cluster setup. i added the lines --jars /path/to/sparklens-0.3.2-s_2.11.jar --conf spark.extraListeners=com.qubole.sparklens.QuboleJobListener in my spark-submit query.
the result was that the executor pods keep on running INFINITELY. they never stop and always stay in "Running" status.
link to kubectl logs
Update: this bug seems to happen with Spark-Pi example as well :\