How to print out Spark connection of Spark session ?

Question

Suppose I run pyspark command and got global variable spark of type SparkSession. As I understand, this spark holds a connection to the Spark master. Can I print out the details of this connection including the hostname of this Spark master ?

Answer 1

For basic information you can use master property:

spark.sparkContext.master

To get details on YARN you might have to dig through hadoopConfiguration:

hadoopConfiguration = spark.sparkContext._jsc.hadoopConfiguration()
hadoopConfiguration.get("yarn.resourcemanager.hostname")

or

hadoopConfiguration.get("yarn.resourcemanager.address")

When submitted to YARN Spark uses Hadoop configuration to determine the resource manger so these values should match ones present in configuration placed in HADOOP_CONF_DIR or YARN_CONF_DIR.