Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How do I control a hive job name but keep the stage info?

Tags:

hadoop

hive

I have a number of hive queries that my system executes on a regular basis. When you look at the job tracker, they show up as "SELECT field, other_field ..... (Stage-1)" and similar. That's not particularly helpful to me, so I added:

set mapred.job.name = more helpful name;
to the query. Now I can tell them apart better. However, now my queries that get split into multiple stages all show up as the same name. What I'd ideally like is something along the lines of

set mapred.job.name = more helpful name (Stage-%d);
where the %d would get replaced by the current stage number.
Is this possible, and does anyone know how?

like image 592
Some Guy Avatar asked Sep 26 '13 18:09

Some Guy


1 Answers

I'm not sure there is a way to implement exactly what you wish but I can offer something else.
Instead of using set mapred.job.name you can add a comment in the beginning of the query with a more helpful name like this :
-- this is a more helpful name
SELECT field, other_field ....

Then, in the jobtracker you'll see -- this is a more helpful name ..... (Stage-%d)"

like image 83
dimamah Avatar answered Oct 11 '22 02:10

dimamah