Menu
I know I can use SparkConf.set('spark.app.name',...) to set appName before creating the SparkContext.
However, I want to change the name of the application as it progresses, i.e., after SparkContext has been created.
Alas, setting sc.appName does not change how the job is shown by yarn application -list.
Is there a way?
494
it returns "true". Hence, it seems like stopping a session stops the context as well, i. e., the second command in my first post is redundant. Please note that in Pyspark isStopped does not seem to work: "'SparkContext' object has no attribute 'isStopped'".
As a result, when comparing SparkSession vs SparkContext, as of Spark 2.0. 0, it is better to use SparkSession because it provides access to all of the Spark features that the other three APIs do.
SparkSession vs SparkContext – Since earlier versions of Spark or Pyspark, SparkContext (JavaSparkContext for Java) is an entry point to Spark programming with RDD and to connect to Spark Cluster, Since Spark 2.0 SparkSession has been introduced and became an entry point to start programming with DataFrame and Dataset.
This is not possible: any update to the sparkConf, including spark.app.name, is only taken into account before the instance of SparkConf is used to instanciate a SparkContext:
Note that once a SparkConf object is passed to Spark, it is cloned and can no longer be modified by the user. Spark does not support modifying the configuration at runtime.
https://spark.apache.org/docs/1.3.1/api/scala/index.html#org.apache.spark.SparkConf
196
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
