How to check status of Spark applications from the command line?

Question

To check running applications in Apache spark, one can check them from the web interface on the URL:

http://<master>:8080

My question how we can check running applications from terminal, is there any command that returns applications status?

Answer 1

If it's for Spark Standalone or Apache Mesos cluster managers, @sb0709's answer is the way to follow.

For YARN, you should use yarn application command:

$ yarn application -help usage: application  -appStates <States>             Works with -list to filter applications                                  based on input comma-separated list of                                  application states. The valid application                                  state can be one of the following:                                  ALL,NEW,NEW_SAVING,SUBMITTED,ACCEPTED,RUN                                  NING,FINISHED,FAILED,KILLED  -appTypes <Types>               Works with -list to filter applications                                  based on input comma-separated list of                                  application types.  -help                           Displays help for all commands.  -kill <Application ID>          Kills the application.  -list                           List applications. Supports optional use                                  of -appTypes to filter applications based                                  on application type, and -appStates to                                  filter applications based on application                                  state.  -movetoqueue <Application ID>   Moves the application to a different                                  queue.  -queue <Queue Name>             Works with the movetoqueue command to                                  specify which queue to move an                                  application to.  -status <Application ID>        Prints the status of the application. 

Answer 2

You can use spark-submit --status (as described in Mastering Apache Spark 2.0).

spark-submit --status [submission ID] 

See the code of spark-submit for reference:

if (!master.startsWith("spark://") && !master.startsWith("mesos://")) {   SparkSubmit.printErrorAndExit(     "Requesting submission statuses is only supported in standalone or Mesos mode!") }