Spark - How can get the Logical / Physical Query execution using - Thirft - Hive Interactor

Question

Spark - How can get the Logical / Physical Query execution using the following

1. Via Thrift
2. Via SparkInteractor

Answer 1

You can use explain statement with query as below in beeline via thrift.

EXPLAIN EXTENDED select * from sr23 join sr12 [<join condidtion>]

What do you mean be spark interceptor.? is it spark-sql shell.? if it is, then you can use above query.

If you meant spark-shell, then you need to call explain() function on dataframes.

eg:

val df1 = sqlContext.sql(" < your sql query > ");

df1.explain;

this will give both physical and logical plans. You can also see them from spark web UI in SQL tab.