Shutdown Hook for spark batch application

Question

I have a spark scala batch application. It commits the run status to mariadb when it completes or fails. I want to implement an edge case when the application is killed by say "yarn application -kill [appid]", I want to update the status as failed in mariadb table.

I planned to use "ShutdownHookManager" for the same but I see it is private in spark and the scala sys.ShutdownHookThread does not work as well.

Can somebody guide me on the shutdown hook handling of killing spark batch application. Not much resources on the same.

Answer 1

You can create a custom SparkListener that reacts on the onApplicationEnd event:

class MyListener extends SparkListener {
  override def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd): Unit = {
    println("Shutting down...")
  }
}

This listener can then be added to the SparkContext:

spark.sparkContext.addSparkListener(new MyListener())

When the Spark application terminates, the string Shutting down... is printed on the console.