Deploy spark driver application without spark submit

Question

Let's suppose we have a spark driver program written like this:

public class SimpleApp {
  public static void main(String[] args) {
    String logFile = "YOUR_SPARK_HOME/README.md"; // Should be some file on your system
    SparkConf conf = new SparkConf().setAppName("Simple Application");
    JavaSparkContext sc = new JavaSparkContext(conf);
    JavaRDD<String> logData = sc.textFile(logFile).cache();

    long numAs = logData.filter(new Function<String, Boolean>() {
      public Boolean call(String s) { return s.contains("a"); }
    }).count();

    long numBs = logData.filter(new Function<String, Boolean>() {
      public Boolean call(String s) { return s.contains("b"); }
    }).count();

    System.out.println("Lines with a: " + numAs + ", lines with b: " + numBs);
  }
}

and I want to run in a yarn cluster, can I avoid using spark-submit and (supposing of course I have access to one cluster node ), by just specifying in the context I want to run on yarn? In other words, is it possible to launch the spark client as a regular java app leveraging yarn?

Answer 1

Here is another official way to do it.

Spark Launcher - Library for launching Spark applications.

This library allows applications to launch Spark programmatically. There's only one entry point to the library - the SparkLauncher class.

To launch a Spark application, just instantiate a SparkLauncher and configure the application to run. For example:

 import org.apache.spark.launcher.SparkLauncher;

   public class MyLauncher {
     public static void main(String[] args) throws Exception {
       Process spark = new SparkLauncher()
         .setAppResource("/my/app.jar")
         .setMainClass("my.spark.app.Main")
         .setMaster("local")
         .setConf(SparkLauncher.DRIVER_MEMORY, "2g")
         .launch();
       spark.waitFor();
     }
   }

You can set all the YARN specific config using setConf method and set the master to yarn-client or yarn-cluster

References: https://spark.apache.org/docs/1.4.0/api/java/org/apache/spark/launcher/package-summary.html