Logo Questions Linux Laravel Mysql Ubuntu Git Menu
HTML CSS JAVASCRIPT SQL PYTHON PHP BOOTSTRAP JAVA JQUERY R React Kotlin
 

How to disable SparkUI programmatically?

Tags:

apache-spark

I noticed that this bug has been fixed:

Allow users to disable Jetty Spark UI in local mode

https://issues.apache.org/jira/browse/SPARK-2100

but how to disable it programmatically?

is it by

sparkContext.setLocalProperty(?, ?)

I checked the documentation, but couldn't figure out which property to set

http://spark.apache.org/docs/latest/configuration.html#spark-ui

like image 482
Alex Luya Avatar asked Nov 18 '15 07:11

Alex Luya

People also ask

How do I keep the Spark session alive?

Solution to your Answer Before SparkContext to stop, use Thread. sleep(86400000). This will keep 24 hours active of your Spark UI until you kill the process.

What is Spark executor heartbeatInterval?

While spark. executor. heartbeatInterval is the interval at executor reports its heartbeats to driver.

What is SparkContext?

A SparkContext represents the connection to a Spark cluster, and can be used to create RDDs, accumulators and broadcast variables on that cluster. Only one SparkContext should be active per JVM.

How do I check my default Spark settings?

In Spark/PySpark you can get the current active SparkContext and its configuration settings by accessing spark. sparkContext. getConf.

1 Answers

Set spark.ui.enabled to false. This configuration isn't documented on Spark's configuration page because it was added as an internal configuration for debugging and unit tests [1].

With spark-submit, you could simply --conf spark.ui.enabled=false.

[1] https://github.com/apache/spark/pull/2363/files#r17436782

like image 169
Josh Rosen Avatar answered Sep 27 '22 17:09

Josh Rosen
Sign in to Comment

Related questions

What is an optimized way of joining large tables in Spark SQL
Where is the reference for options for writing or reading per format?
Spark SQL nested withColumn
Spark 1.5.2: org.apache.spark.sql.AnalysisException: unresolved operator 'Union;
PySpark & MLLib: Random Forest Feature Importances
Distributed Web crawling using Apache Spark - Is it Possible?
What is rank in ALS machine Learning Algorithm in Apache Spark Mllib
Spark - Creating Nested DataFrame
spark sql current timestamp function
Spark 2.0: Relative path in absolute URI (spark-warehouse)
spark dataframe groupby multiple times
How to execute spark submit on amazon EMR from Lambda function?
How to import pyspark in anaconda
Convert comma separated string to array in pyspark dataframe
Spark on YARN resource manager: Relation between YARN Containers and Spark Executors
How do I convert a WrappedArray column in spark dataframe to Strings?
Use collect_list and collect_set in Spark SQL
Rename nested field in spark dataframe
Scala add new column to dataframe by expression
How to read multiple gzipped files from S3 into a single RDD?

Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!