How oozie handle dependencies?

Question

I have several questions about oozie 2.3 share libraries:

Currently, I defined the share libraries in our coordinator.properties:

oozie.use.system.libpath=true 
oozie.libpath=<hdfs_path>

Here are my questions:

1. When share libraries are copied to other data node and how many data node will get share libraries?

2. Are the share libraries copied to other data node based on number of wf in a coordinator job or they are only copied once per coordinator job?

Answer 1

Adding entries to the oozie.libpath property effectively means that OOZIE will configure those libraries to be in the mapred.cache.files configuration property (this is a DistributedCache property) when the actions in your workflow are executed.

Hadoop will then take care of copying those jars to each cluster node once per job, and the tasks are then configured with the jar in the classpath configuration property mapred.job.classpath.files

So in response to your second question, they will be copied over for each action in the workflow, not once per coordinator job. So if you have a wf job that has 4 mapreduce actions, the libraries will be copied to each tasktracker (only those task trackers that participate in the mapreduce job) 4 times in the lifetime of that workflow.