Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Can I rename the oozie job name dynamically

Tags:

hadoop

oozie

We have a Hadoop service in which we have multiple applications. We need to process the data for each of the applications by reexecuting the same workflow. These are scheduled to execute at the same time of the day. The issue is that when these jobs are running its hard to know for which application the job is running/failed/succeeded. Ofcourse, I can open the job coonfiguration and know it but that does take time since there are 10s of applications running under that service.

Is there any option in oozie to dynamically pass the name of the workflow (or part of it) when executing the job such as

oozie job -run -config <filename> -name "<NameIWishToGive>"
OR
oozie job -run -config <filename> -nameSuffix "<MyApplicationNameUnderTheService>"

Also, we dont wish to create multiple job folders to execute separately as that would be too much of copy paste.

Please suggest.

like image 384
explorer Avatar asked Jan 28 '13 14:01

explorer


People also ask

What is prep status in Oozie?

Oozie then creates a record for the bundle with status PREP and returns a unique ID. When a user requests to suspend a bundle job that is in PREP state, oozie puts the job in status PREPSUSPENDED . Similarly, when pause time reaches for a bundle job with PREP status, oozie puts the job in status PREPPAUSED .

What is coordinator in Oozie?

The Oozie Coordinator system allows the user to define and execute recurrent and interdependent workflow jobs (data application pipelines). Real world data application pipelines have to account for reprocessing, late processing, catchup, partial processing, monitoring, notification and SLAs.

Which language is used for the core syntax of Oozie?

The workflow definition language is XML based and it is called hPDL (Hadoop Process Definition Language). Refer to the Appendix A for theOozie Workflow Definition XML Schema .


2 Answers

It looks to me like you should be able to just use properties set in the job config.

I was able to get a dynamic name by doing the following.

Here's an example of my workflow.xml:

<workflow-app xmlns="uri:oozie:workflow:0.2" name="map-reduce-wf-${environment}">
...
</workflow-app>

And in my job.properties I had:

...
environment=test
...

The name ended up being: "map-reduce-wf-test"

like image 106
Mike Barker Avatar answered Sep 24 '22 14:09

Mike Barker


you will find a whole bunch of oozie command lines here in the apache docs. i'm not sure which one exactly you are looking for so i thought i'd just paste the link. hope this helps!

like image 45
Rachel Gallen Avatar answered Sep 21 '22 14:09

Rachel Gallen