sparklyr / sparklyr

R interface for Apache Spark

Home Page:https://spark.rstudio.com/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Correctly shutdown on YARN cluster for deploy mode "cluster"

smingerson opened this issue · comments

This sparklyr line calls System.exit(0) for non-service connections. However, according to this comment in the YARN ApplicationMaster code, it is incorrect for System.exit(0) (or status) to be called by the ending spark context.

Failure to exit correctly leads to a reported status of "FAILED" by the YARN ApplicationMaster, which can lead to spurious re-attempts of the submitted job.

To fix, I suggest updating the conditional logic so it is

if (!isService && !isBatch) System.exit(0)