Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Hadoop java.io.IOException: Mkdirs failed to create /some/path

When I try to run my Job I am getting the following exception:

Exception in thread "main" java.io.IOException: Mkdirs failed to create /some/path
    at org.apache.hadoop.util.RunJar.ensureDirectory(RunJar.java:106)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:150)

Where the /some/path is hadoop.tmp.dir. However when I issue the dfs -ls cmd on /some/path I can see that it exists and the dataset file is present (was copied before lunching the job). Also the path is correctly defined in hadoop configs. Any suggestions will be appreciated. I am using hadoop 0.21.

like image 244
alien01 Avatar asked May 09 '12 19:05

alien01


3 Answers

Just ran into this problem running mahout from CDH4 in standalone mode in my MacBook Air.

The issue is that a /tmp/hadoop-xxx/xxx/LICENSE file and a /tmp/hadoop-xxx/xxx/license directory are being created on a case-insensitive file system when unjarring the mahout jobs.

I was able to workaround this by deleting META-INF/LICENSE from the jar file like this:

zip -d mahout-examples-0.6-cdh4.0.0-job.jar META-INF/LICENSE

and then verified it with

jar tvf mahout-examples-0.6-cdh4.0.0-job.jar | grep -i license

Hope this helps!

like image 68
Todd Nemet Avatar answered Nov 13 '22 10:11

Todd Nemet


The problem is OSX specific it is due to the fact that by default the filesystem is set to case-insensitive on a Mac (case preserving but case insensitive, which to my opinion is very bad).

A hack to circumvent this is to create a .dmg disk image with disk utility which is case sensitive and mount this image where you need it (i.e. hadoop.tmp.dir or /tmp) with the following command (as a superuser):

sudo hdiutil attach -mountpoint /tmp <my_image>.dmg

I hope it helps.

like image 24
ngrislain Avatar answered Nov 13 '22 11:11

ngrislain


This is a file on the local disk that is being created (to unpack your job jar into), not in HDFS. Check you have permissions to mkdir this directory (try it from the command line)

like image 12
Chris White Avatar answered Nov 13 '22 10:11

Chris White