Menu
I am new to spark.In my current spark application script, I can send queries to spark in-memory saved table and getting the desired result using spark-submit.The problem is, each time spark context stops automatically after completing result. I want to send multiple queries sequentially.for that I need to keep alive spark context. how could I do that ? my point is
Manual start and stop sparkcontext by user
kindly suggest me.I am using pyspark 2.1.0.Thanks in advance
289
You can stop the SparkContext by calling the stop() method. As explained above you can have only one SparkContext per JVM. If you wanted to create another, you need to shutdown it first by using stop() method and create a new SparkContext.
To create a SparkContext you first need to build a SparkConf object that contains information about your application. SparkConf conf = new SparkConf(). setAppName(appName). setMaster(master); JavaSparkContext sc = new JavaSparkContext(conf);
You should stop() the active SparkContext before creating a new one. The Spark driver program creates and uses SparkContext to connect to the cluster manager to submit Spark jobs, and know what resource manager (YARN, Mesos or Standalone) to communicate to. It is the heart of the Spark application.
To answer your question, this works
import pyspark
# start
sc = pyspark.SparkContext()
#stop
sc.stop()
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
