What's the difference between spark-shell and spark-sql? Any performance differences?

Question

Spark-shell: which basically opens the scala> prompt. Where query needs to write in below manner

val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
// Queries are expressed in HiveQL
sqlContext.sql("FROM src SELECT key, value").collect().foreach(println)

spark-sql: Which seems to connect directly to hive metastore and we can write a query in a similar way to hive. And query over existing data in hive

I want to know the difference between these two.. And does processing any query in spark-sql is same as in spark-shell? I mean can we leverage the performance benefits of spark in spark-sql?

Spark 1.5.2's here.

Answer 1

spark-shell gives you a working Spark environment where Scala is the (programming) language.

spark-sql gives you a Spark SQL environment where SQL is the query language.

Note that spark-shell is for any APIs available in Spark while spark-sql is only for Spark SQL API (with Datasets and DataFrames).

They're simply different interfaces for users with different skills (spark-shell for Spark/Scala developers while spark-sql for SQL developers).

spark-sql "hides" the Spark infrastructure behind SQL interface which places it higher in how much engineering skills one should have, but eventually uses all the optimizations available in Spark SQL (and Spark in general).

Performance-wise spark-sql and spark-shell are alike.