Correctly shutdown on YARN cluster for deploy mode "cluster"
smingerson opened this issue · comments
This sparklyr line calls System.exit(0)
for non-service connections. However, according to this comment in the YARN ApplicationMaster code, it is incorrect for System.exit(0)
(or status) to be called by the ending spark context.
Failure to exit correctly leads to a reported status of "FAILED" by the YARN ApplicationMaster, which can lead to spurious re-attempts of the submitted job.
To fix, I suggest updating the conditional logic so it is
if (!isService && !isBatch) System.exit(0)