Why is SparkListenerApplicationStart never fired?

Question

I am writing a Spark application and I need to intercept the status of the running jobs. I implemented a SparkListener for this purpose, using the following code:

class MyAppListener extends SparkListener {

    override def onApplicationStart(ev: SparkListenerApplicationStart): Unit = {
      println("AAA: Application Start")
    }

    override def onApplicationEnd(ev: SparkListenerApplicationEnd): Unit = {
      println("AAA: Application End")
    }
  }
}

Then, I used the following code to start the application and see the events:

val appListener = new MyAppListener
val conf = new SparkConf().setAppName("Listener")
val sc = new SparkContext(conf) 
sc.addSparkListener(appListener)
println(sc.parallelize(1 to 10).count)
sc.stop()

In the logs, I see the string "AAA: Application End", but I don't see the start of the application.

Configuration:

• Spark version 1.2.0
• Scala 2.10
• local standalone mode or YARN client mode on a cluster (same behavior)

Answer 1

You were adding your listener to spark in the wrong place, When you initiate a spark context, it also starts your application.=> At the point you added your listener, onApplicationStart has already been fired.

Solution: Add your listener to SparkConf.

sparkConf.set("spark.extraListeners","your.listener.class")