Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Setting additional classpath for a hadoop tool

How can I dynamically set up an additional classpath so that jars are accessible from a hadoop tool class? I'm running my job via "hadoop jar". I need something like the "-libjars" switch which unfortunately works only for mappers and reducers, and my Tool implementations does not see jars provided by "libjars"

export "HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/path/to/my.jar && hadoop jar ..." seems making "my.jar" accessible to the Tool, but then libraries that are inside the job jar in the "/lib" folder become unaccessible.

like image 245
Vyacheslav Avatar asked Nov 04 '22 07:11

Vyacheslav


1 Answers

Additionally, if you have conflicts with some jars included into a Hadoop distribution, you can export HADOOP_USER_CLASSPATH_FIRST=true. Maybe will be useful to somebody.

like image 74
Vyacheslav Avatar answered Nov 07 '22 22:11

Vyacheslav