How to get WebUI URI from SparkContext

Question

I am running pySpark in an IPython notebook. Each notebook has its own sparkContext, and each one has an associated webUI on its own port, starting from 4040.
I would like to extract and print the port or the URI, so that I can browse to the correct webUI. How can I get this inormation ?

Answer 1

You can get the full URL the the SparkUI page by tunnelling straight to the underlying Scala SparkContext with:

>>> sc._jsc.sc().uiWebUrl().get()
u'http://192.168.0.59:6970'

This is a bit inconvenient, however, so I've just filed a Pull Request that adds an accessor, so you can simply do:

>>> sc.uiWebUrl
u'http://192.168.0.59:6970'

Hopefully this will be merged into the next release, but if not, you can simply patch your own copy of Spark with the changes in the linked branch (or use the uglier long form above).

EDIT: The PR has indeed been merged, so the sc.uiWebUrl syntax above will now work on the latest versions of Spark without any patching.