How to check that the SparkContext has been stopped?

Question

How do I detect whether the SparkContext has been stopped?

Answer 1

This applies to the Scala/Java API for that writing time

Before Spark has released version 1.6, you wouldn't be able to check it, but only to trigger it:

sc.stop()

From version 1.6 and above, you have a boolean function that returns true if context is stopped or in the midst of stopping:

sc.isStopped

This applies to PySpark API

Thanks for @zero323 comment:

sc._jsc.sc().isStopped()

Which gives you the Java SparkContext.

Answer 2

If you use spark 1.5, that could be done via reflection API:

boolean isStopped(SparkContext sc) throws NoSuchFieldException, IllegalAccessException {
    Field f = sc.getClass().getDeclaredField("stopped"); 
    f.setAccessible(true);
    AtomicBoolean stopped = (AtomicBoolean) f.get(sc);
    return stopped.get();
}