How to Access RDD Tables via Spark SQL as a JDBC Distributed Query Engine?

Question

Several postings on stackoverflow has responses with partial information about How to Access RDD Tables via Spark SQL as a JDBC Distributed Query Engine. So I'd like to ask the following questions for complete information about how to do that:

1. In the Spark SQL app, do we need to use HiveContext to register tables? Or can we use just SQL Context?

2. Where and how do we use HiveThriftServer2.startWithContext?

3. When we run start-thriftserver.sh as in

/opt/mapr/spark/spark-1.3.1/sbin/start-thriftserver.sh --master spark://spark-master:7077 --hiveconf hive.server2.thrift.bind.host spark-master --hiveconf hive.server2.trift.port 10001

besides specifying the jar and main class of the Spark SQL app, do we need to specify any other parameters?

4. Are there any other things we need to do?

Thanks.

Answer 1

To expose DataFrame temp tables through HiveThriftServer2.startWithContext(), you may need to write and run a simple application, may not need to run start-thriftserver.sh.

To your questions:

1. HiveContext is needed; the sqlContext converted to HiveContext implicitly in spark-shell

2. Write a simple application, example :

    import  org.apache.spark.sql.hive.thriftserver._  
    val  hiveContext  =  new  HiveContext(sparkContext)
    hiveContext.parquetFile(path).registerTempTable("my_table1")
      HiveThriftServer2.startWithContext(hiveContext)

3. No need to run start-thriftserver.sh, but run your own application instead, e.g.:

spark-submit --class com.xxx.MyJdbcApp ./package_with_my_app.jar

4. Nothing else from server side, should start on default port 10000; you may verify by connecting to the server with beeline.