Menu
I am using Spark on my server with Ubuntu. I use the shell only and there is no UI for the server other than that. There is always an option of looking at the stats of Spark like jobs, cpus, memory etc when working on local computer on a web browser.
Is there a way that I can view all that information locally on the server?
895
Open http://localhost:18080 in your browser to view the Spark UI locally.
Apache Spark provides a suite of web user interfaces (UIs) that you can use to monitor the status and resource consumption of your Spark cluster.
Solution to your Answer Before SparkContext to stop, use Thread. sleep(86400000). This will keep 24 hours active of your Spark UI until you kill the process.
The web UI is accessible in Databricks by going to "Clusters" and then clicking on the "View Spark UI" link for your cluster, it is also available by clicking at the top left of this notebook where you would select the cluster to attach this notebook to. In this option will be a link to the Apache Spark Web UI.
spark-shell always starts a web UI on port 4040 by default. If the server has no graphical interface, you can either use a text-based browser, such as Links, or access port 4040 from another machine, such as your laptop. You may need to run ufw allow 4040 to unblock the port in the firewall.
133
I think a python should be running on the server, so you can use this command:
python -m webbrowser -n http://host:port
(for example the application detail ui: http://localhost:4040). It gives a shell based browser if the server has no graphical interface
30
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
