YARN Application exited with exitCode: -1000 Not able to initialize user directories

Question

I am getting:

Application application_1427711869990_0001 failed 2 times due to AM Container for appattempt_1427711869990_0001_000002 exited with exitCode: -1000 due to: Not able to initialize user directories in any of the configured local directories for user kailash
.Failing this attempt.. Failing the application.

I couldn`t find anything related to this exit code and the associated reason. I am using Hadoop 2.5.0 (Cloudera 5.3.2).

Answer 1

Actually this is due to the permission issues on some of the yarn local directories. I started using LinuxContainerExecutor (in non secure mode with nonsecure-mode.local-user as kailash) and made corresponding changes. However due to some (unknown) reason NodeManager failed to clean local directories for users, and there still existed directories with previous user (in my case yarn).

So to solve this, I first had to find the value of the property yarn.nodemanager.local-dirs (with Cloudera use search option to find this property for YARN service, otherwise look into yarn-site.xml in hadoop conf directory), and then delate the files/directories under usercache for all the node manager nodes. In my case, I used:

rm -rf /yarn/nm/usercache/*

Answer 2

Just in case if someone is lost with usercache location. If you don't have the yarn.nodemanager.local-dirs configured anywhere, look for it in the default location ${hadoop.tmp.dir}/nm-local-dir. Again, if the hadoop.tmp.dir is not configured in the core-site.xml, it will be under /tmp/hadoop-${user.name}. The user.name is the UNIX user that you are using to run the current Hadoop process. All the configuration file are under $HADOOP_INSTALL/etc/hadoop/ by default.