Menu
I'd like, from where I run a Spark job, to get the unique id of that job.
Via the Spark master node website, I can see that id. It's something like:
ID: app-20140429125304-0452
Is there any way to get this, when creating a running a job? Maybe via the SparkContext?
320
Is there any way to get the Spark Job Id that is associated with a running Task (preferably using TaskContext or SparkEnv)? @FRG96 (Customer) , Honestly, there is no easy way to do this. Your only option is to set up cluster log delivery, which will give you access to the cluster's event log file.
applicationId. A unique identifier for the Spark application. Its format depends on the scheduler implementation. in case of local spark app something like 'local-1433865536131' in case of YARN something like 'application_1433865536131_34483'
Also how to setup debug mode in spark-shelI. You can pass your own "log4j. properties" path to log messages and pass it to your spark shell command. Then run the spark-shell as following then you should see DEBUG messages.
yes, exactly as you said:
sc.applicationId
res0: String = app-20150224184813-11531
(This is spark 1.2)
See API doc at
187
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
