Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to add custom description to Spark Job for displaying in Spark Web UI

Tags:

When we submit application to Spark, and after performing any operation Spark Web UI displays Job and Stages like count at MyJob.scala:15. But in my application there are multiple count and save operations are there. So it is very difficult to understand UI. Instead of count at MyJob.scala:15, can we add custom description to give more detailed information to job.

While googling found https://issues.apache.org/jira/browse/SPARK-3468 and https://github.com/apache/spark/pull/2342, author attached image, with detailed description like 'Count', 'Cache and Count', 'Job with delays'. So can we achieve same? I am using Spark 2.0.0.

like image 575
Sheel Avatar asked Aug 24 '16 12:08

Sheel


2 Answers

use the sc.setJobGroup:

Examples:
python:

In [28]: sc.setJobGroup("my job group id", "job description goes here") In [29]: lines = sc.parallelize([1,2,3,4]) In [30]: lines.count() Out[30]: 4 

Scala:

scala> sc.setJobGroup("my job group id", "job description goes here") scala> val lines = sc.parallelize(List(1,2,3,4)) scala> lines.count() res3: Long = 4 

SparkUI:

job description screen shot

I hope this is what you are looking for.

like image 115
Ajeet Shah Avatar answered Sep 19 '22 18:09

Ajeet Shah


Note that new Zeppelin 0.8 loose his tracking hook if you change the JobGroup name, and can't display his job progress bar (job still working, no effect on the job itself)

You can use

sc.setLocalProperty("callSite.short","my job description") sc.setLocalProperty("callSite.long","my job details long description") 

instead

See How to change job/stage description in web UI? for some screen captures and scala syntax

like image 45
Ftagn Avatar answered Sep 18 '22 18:09

Ftagn