Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to pass multiple statements into Spark SQL HiveContext

For example I have few Hive HQL statements which I want to pass into Spark SQL:

set parquet.compression=SNAPPY;
create table MY_TABLE stored as parquet as select * from ANOTHER_TABLE;
select * from MY_TABLE limit 5;

Following doesn't work:

hiveContext.sql("set parquet.compression=SNAPPY; create table MY_TABLE stored as parquet as select * from ANOTHER_TABLE; select * from MY_TABLE limit 5;")

How to pass the statements into Spark SQL?

like image 352
Andrey Dmitriev Avatar asked Apr 29 '16 12:04

Andrey Dmitriev


2 Answers

I worked on a scenario where i needed to read a sql file and run all the; separated queries present in that file.

One simple way to do it is like this:

val hsc = new org.apache.spark.sql.hive.HiveContext(sc)
val sql_file = "/hdfs/path/to/file.sql"
val file = sc.wholeTextFiles(s"$sql_file")
val queries = f.take(1)(0)._2
Predef.refArrayOps(queries.split(';')).map(query => hsc.sql(query))
like image 74
kuldeep singh Avatar answered Nov 12 '22 21:11

kuldeep singh


Thank you to @SamsonScharfrichter for the answer.

This will work:

hiveContext.sql("set spark.sql.parquet.compression.codec=SNAPPY")
hiveContext.sql("create table MY_TABLE stored as parquet as select * from ANOTHER_TABLE")
val rs = hiveContext.sql("select * from MY_TABLE limit 5")

Please note that in this particular case instead of parquet.compression key we need to use spark.sql.parquet.compression.codec

like image 34
Andrey Dmitriev Avatar answered Nov 12 '22 22:11

Andrey Dmitriev