How to invoke spark job in context of REST Web-service?

Question

I want to run Spark SQL queries in my restful web service, So How can I run Spark Context via Jersey context? I need to pass my Spark SQL request to the cluster then return the result to the user via REST API. But in Spark Documentations, there is no way to run Spark SQL Queries in java code without submitting the jar file to the cluster (master/slaves).

Answer 1

If you are using Spark version > 1.4 then you can use SparkLauncher to run your application.

import org.apache.spark.launcher.SparkLauncher;

public class MyLauncher {
  public static void main(String[] args) throws Exception {
    Process spark = new SparkLauncher()
      .setAppResource("/my/app.jar")
      .setMainClass("my.spark.app.Main")
      .setMaster("local")
      .setConf(SparkLauncher.DRIVER_MEMORY, "2g")
      .launch();
    spark.waitFor();
  }
}

In order for it to work you should give it a jar file. Since you want to run a SparkSQL query then you could pack it in a single jar file or you could have a jar getting as parameter the query you want to execute.

The caveat is that you have to start-stop the SparkContext every time you want to execute the query. If you don't mind waiting for it then it is fine. But if time is a big thing then I would recommend writing a separate service that would have the spark context always up and your application would make calls to it.