How do I get a spark dataframe to print it's explain plan to a string

Question

I have a system in which component A passes sql to component B, B then runs the sql via apache spark, and returns a result.

For debugging purposes, I'm putting in a second communication channel where A can pass sql to B and request and explaination of the plan.

The code in B looks something like this:

def handleExplain(sql: String, extended: Boolean): String = {
    val dataFrame = sparkContext.sql(sql)
    dataFrame.explain(extended)
}

The problem is that 'explain' doesn't return a string, it just prints the explain plan to the console. How do I get the string contents of what's printed to the console? Is there another function, or do I have to lift it from the console?

Answer 1

All query plans, including string representation, can be accessed through corresponding QueryExecution object. For example to get full execution plan:

val ds: Dataset[_] = ???

ds.queryExecution.toString

only logical plan:

ds.queryExecution.logical.toString

optimized logical plan:

ds.queryExecution.optimizedPlan.toString

or executed / physical plan:

ds.queryExecution.executedPlan