How to use the programmatic spark submit capability

Question

There is a somewhat recent (Spring 2015) feature apparently intended to allow submitting a spark job programmatically.

Here is the JIRA https://issues.apache.org/jira/browse/SPARK-4924

However there is uncertainty (and count me as well) about how to actually use these features. Here are the last comments in the jira:

When asking the actual author of this work to further explain it is "look in the API docs".

The "user document" is the Spark API documentation.

The author did not provide further details and apparently feels the whole issue were self explanatory. If anyone can connect the dots here: specifically - where in the API docs is this newer Spark Submit capability described - it would be appreciated.

Here is some of the info I am looking for -Pointers to the following:

• What capabilities have been added to the spark api
• How do we use them
• Any examples / other relevant documentation and/or code

Update The SparkLauncher referred to in the accepted answer does launch a simple app under trivial ( master=local[*]) conditions. It remains to be seen how usable it will be on an actual cluster. After adding a print statement to the linked code:

println("launched.. and waiting..") spark.waitFor()

We do see:

launched.. and waiting..

Well this is probably a small step forward. Will update this question as I move towards a real clustered environment.

Answer 1

Looking at the details of the pull request, it seems that the functionality is provided by the SparkLauncher class, described in the API docs here.

public class SparkLauncher extends Object

Launcher for Spark applications.

Use this class to start Spark applications programmatically. The class uses a builder pattern to allow clients to configure the Spark application and launch it as a child process.

The API docs are rather minimal, but I found a blog post that gives a worked example (code also available in a GitHub repo). I have copied a simplified version of the example below (untested) in case the links go stale:

import org.apache.spark.launcher.SparkLauncher

object Launcher extends App {
  val spark = new SparkLauncher()
    .setSparkHome("/home/user/spark-1.4.0-bin-hadoop2.6")
    .setAppResource("/home/user/example-assembly-1.0.jar")
    .setMainClass("MySparkApp")
    .setMaster("local[*]")
    .launch();
  spark.waitFor();
}

See also:

• Another tutorial blog post / review of the feature
• A book chapter on the topic