Accessing Spark RDDs from a web browser via thrift server - java

Question

We have processed our data using Spark 1.2.1 with Java and stored in Hive tables. We want to access this data as RDDs from an web browser.

I read documentation and I understood the steps to do the task.

I am unable to find the way to interact with Spark SQL RDDs via thrift server. Examples I found have belw line in the code and I am not find the class for this in Spark 1.2.1 java API docs.

HiveThriftServer2.startWithContext

In github i saw scala examples using import org.apache.spark.sql.hive.thriftserver , but I dont see this in Java API docs. Not sure if I am missing something.

Did anybody had luck with accessing Spark SQL RDDs from a browser via thrift? Can you post the code snippet. We are using Java.

Answer 1

I've got most of this working. Lets dissect each part of it: (References at bottom of post)

HiveThriftServer2.startWithContext is defined in Scala. I was never able to access it from Java or from Python using Py4j, and am no JVM expert, but I ended up switching to Scala. This may have something to do with the annotation @DeveloperApi . This is how I imported it Scala in Spark 1.6.1:

import org.apache.spark.sql.hive.thriftserver.HiveThriftServer2

For anyone reading this and not using Hive, a Spark SQL context won't do, and you need a hive context. However, the HiveContext constructor requires a Java spark context, not a scala one.

import org.apache.spark.api.java.JavaSparkContext
import org.apache.spark.sql.hive.HiveContext

var hiveContext = new HiveContext(JavaSparkContext.toSparkContext(sc))

Now start the thrift server

HiveThriftServer2.startWithContext(hiveContext)

// Yay

Next, we need to make our RDDs available as SQL tables. First, we have to convert them into Spark SQL DataFrames:

val someDF = hiveContext.createDataFrame(someRDD)

Then, we need to turn them into Spark SQL tables. You do this by persisting them to Hive, or making the RDD available as a temporary table.

Persist to Hive:

// Deprecated since Spark 1.4, to be removed in Spark 2.0:
someDF.saveAsTable("someTable")

// Up-to-date at time of writing
someDF.write().saveAsTable("someTable")

Or, use a temporary table:

// Use the Data Frame as a Temporary Table
// Introduced in Spark 1.3.0
someDF.registerTempTable("someTable")

Note - temporary tables are isolated to an SQL session. Spark's hive thrift server is multi-session by default in version 1.6 (one session per connection). Therefore, for clients to access temporary tables you've registered, you'll need to set the option spark.sql.hive.thriftServer.singleSession to true

You can test this by querying the tables in beeline, a command line utility for interacting with the hive thrift server. It ships with Spark.

Finally, you need a way of accessing the hive thrift server from the browser. Thanks to its awesome developers, it has an HTTP mode, so if you want to build a web app, you can use the thrift protocol over AJAX requests from the browser. A simpler strategy might be to create an IPython notebook, and use pyhive to connect to the thrift server.

Data Frame Reference: https://spark.apache.org/docs/1.6.0/api/java/org/apache/spark/sql/DataFrame.html

singleSession option pull request: https://mail-archives.apache.org/mod_mbox/spark-commits/201511.mbox/%[email protected]%3E

HTTP mode and beeline howto: https://spark.apache.org/docs/latest/sql-programming-guide.html#distributed-sql-engine

Pyhive: https://github.com/dropbox/PyHive

HiveThriftServer2 startWithContext definition: https://github.com/apache/spark/blob/6b1a6180e7bd45b0a0ec47de9f7c7956543f4dfa/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2.scala#L56-73