Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Why does SparkContext randomly close, and how do you restart it from Zeppelin?

Tags:

I am working in Zeppelin writing spark-sql queries and sometimes I suddenly start getting this error (after not changing code):

Cannot call methods on a stopped SparkContext.

Then the output says further down:

The currently active SparkContext was created at:

(No active SparkContext.)

This obviously doesn't make sense. Is this a bug in Zeppelin? Or am I doing something wrong? How can I restart the SparkContext?

Thank you

like image 646
The Puma Avatar asked Feb 19 '16 21:02

The Puma


People also ask

How do I restart my Zeppelin interpreter?

You can restart the interpreter for the notebook in the interpreter bindings (gear in upper right hand corner) by clicking on the restart icon to the left of the interpreter in question (in this case it would be the spark interpreter).

How do I start SparkContext?

To create a SparkContext you first need to build a SparkConf object that contains information about your application. The appName parameter is a name for your application to show on the cluster UI. master is a Spark, Mesos or YARN cluster URL, or a special “local” string to run in local mode.

How do you stop SparkContext?

You can stop the SparkContext by calling the stop() method.


1 Answers

I have faced this problem a couple of times.

If you are setting your master as yarn-client, it might be due to the stop / restart of Resource Manager, the interpreter process may still be running but the Spark Context (which is a Yarn application) does not exists any more.

You could check if Spark Context is still running by consulting your Resource manager web Interface and check if there is an application named Zeppelin running.

Sometimes restarting the interpreter process from within Zeppelin (interpreter tab --> spark --> restart) will solve the problem.

Other times you need to:

  • kill the Spark interpreter process from the command line
  • remove the Spark Interpreter PID file
  • and the next time you start a paragraph it will start new spark context
like image 169
user1314742 Avatar answered Sep 23 '22 23:09

user1314742