How to suppress the "Stage 2===>" from the output console in spark?

Question

I have dataframe and trying to get distinct count and able to get distinct count successfully but whenever scala program is executing i'm getting this message ([Stage 2:=============================> (1 + 1) / 2])how can i suppress particular this message in console.

val countID = dataDF.select(substring(col("dataDF"),5,7).distinct().count()

Answer 1

You need to set spark.ui.showConsoleProgress to false

I found this in the comments of the ticket for the addition of the progress bar.

https://issues.apache.org/jira/browse/SPARK-4017

I haven't seen it in any of the documentation though, it really should be added.