How can I dynamically set up an additional classpath so that jars are accessible from a hadoop tool class? I'm running my job via "hadoop jar". I need something like the "-libjars" switch which unfortunately works only for mappers and reducers, and my Tool implementations does not see jars provided by "libjars"
export "HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/path/to/my.jar && hadoop jar ..." seems making "my.jar" accessible to the Tool, but then libraries that are inside the job jar in the "/lib" folder become unaccessible.
Additionally, if you have conflicts with some jars included into a Hadoop distribution, you can export HADOOP_USER_CLASSPATH_FIRST=true
. Maybe will be useful to somebody.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With