Menu
I noticed that this bug has been fixed:
Allow users to disable Jetty Spark UI in local mode
https://issues.apache.org/jira/browse/SPARK-2100
but how to disable it programmatically?
is it by
sparkContext.setLocalProperty(?, ?)
I checked the documentation, but couldn't figure out which property to set
http://spark.apache.org/docs/latest/configuration.html#spark-ui
482
Solution to your Answer Before SparkContext to stop, use Thread. sleep(86400000). This will keep 24 hours active of your Spark UI until you kill the process.
While spark. executor. heartbeatInterval is the interval at executor reports its heartbeats to driver.
A SparkContext represents the connection to a Spark cluster, and can be used to create RDDs, accumulators and broadcast variables on that cluster. Only one SparkContext should be active per JVM.
In Spark/PySpark you can get the current active SparkContext and its configuration settings by accessing spark. sparkContext. getConf.
Set spark.ui.enabled to false. This configuration isn't documented on Spark's configuration page because it was added as an internal configuration for debugging and unit tests [1].
With spark-submit, you could simply --conf spark.ui.enabled=false.
[1] https://github.com/apache/spark/pull/2363/files#r17436782
169
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
