Menu
I run the basic example of Hortonworks' yarn application example. The application fails and I want to read the logs to figure out why. But I can't find any files at the expected location (/HADOOP_INSTALL_FOLDER/logs) where the logs of my mapreduce jobs are stored.
Does anybody know where yarn stores the non-mapreduce log files? Do I have to configure a special directory in the xml files?
405
Application Master logs are stored on the node where the jog runs. Because jobs might run on any node in the cluster, open the job log in the InfoSphere® DataStage® and QualityStage® Designer client and look for messages similar to these messages: Connecting to YARN Application Master at node_name : port_number.
You have to do this on the node where resource manager runs. on other nodes, this directory would be empty only. so this should be good. For Question 3, The log files location can be found out by checking hadoop-env.sh or yarn-env.sh file which are present in HADOOP_CONF_DIR which is usually /etc/hadoop/conf/ .
Try "yarn application -list" to get a list of all Yarn apps, followed by "yarn logs -applicationId" to get logs for a particular app. You can find details about these and other yarn commands here.
The container logs should be under yarn.nodemanager.log-dirs:
Where to store container logs. An application's localized log directory will be found in
${yarn.nodemanager.log-dirs}/application_${appid}. Individual containers' log directories will be below this, in directories namedcontainer_{$contid}. Each container directory will contain the files stderr, stdin, and syslog generated by that container.
121
Log-aggregation has been implemented in YARN, because of which the log file locations will vary when compared with Hadoop 1.
You can get the logs of your application in two ways, WebUi and Command line access.
Please go through the below document which gives you a very clear information on this log-aggregation implementation on YARN.
http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/
Thanks to the blog author, Vinod Kumar Vavilapalli.
34
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
