Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Hadoop 2.3.0 wordcount runs forever

I am trying to test my hadoop installation by running a wordcount job. My problem is that the job gets stuck at ACCEPTED state and seems to run forever. I am using hadoop 2.3.0 and tried fix the problem by following an answer to this question here but it didn't work for me.

This is what I have:

C:\hadoop-2.3.0>yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.3.0.jar wordcount /data/test.txt /data/output
15/03/15 15:36:07 INFO client.RMProxy: Connecting to ResourceManager at/0.0.0.0:8032
15/03/15 15:36:09 INFO input.FileInputFormat: Total input paths to process : 1
15/03/15 15:36:10 INFO mapreduce.JobSubmitter: number of splits:1
15/03/15 15:36:10 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_14 26430101974_0001
15/03/15 15:36:11 INFO impl.YarnClientImpl: Submitted application application_14 26430101974_0001
15/03/15 15:36:11 INFO mapreduce.Job: The url to track the job: http://Agata-PC:8088/proxy/application_1426430101974_0001/
15/03/15 15:36:11 INFO mapreduce.Job: Running job: job_1426430101974_0001

This is my mapred-site.xml:

<configuration>
<property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
</property>
 <property>
    <name>mapred.job.tracker</name>
    <value>127.0.0.1:9001</value>
</property>
   <property>
    <name>mapreduce.jobtracker.staging.root.dir</name>
    <value>/user</value>
</property>
<property>
    <name>mapreduce.history.server.http.address</name>
    <value>127.0.0.1:51111</value>
    <description>Http address of the history server</description>
    <final>false</final>
</property>
<property>
    <name>yarn.app.mapreduce.am.resource.mb</name>
    <value>1024</value>
</property>
<property>
    <name>yarn.app.mapreduce.am.command-opts</name>
    <value>-Xmx768m</value>
</property>
<property>
    <name>mapreduce.map.cpu.vcores</name>
    <value>1</value>
    <description>The number of virtual cores required for each map task.</description>
</property>
<property>
    <name>mapreduce.reduce.cpu.vcores</name>
    <value>1</value>
    <description>The number of virtual cores required for each map task.</description>
</property>
<property>
    <name>mapreduce.map.memory.mb</name>
    <value>1024</value>
    <description>Larger resource limit for maps.</description>
</property>
<property>
    <name>mapreduce.map.java.opts</name>
    <value>-Xmx768m</value>
    <description>Heap-size for child jvms of maps.</description>
</property>
<property>
    <name>mapreduce.reduce.memory.mb</name>
    <value>1024</value>
    <description>Larger resource limit for reduces.</description>
</property>
<property>
    <name>mapreduce.reduce.java.opts</name>
    <value>-Xmx768m</value>
    <description>Heap-size for child jvms of reduces.</description>
</property>
</configuration>

And this is my yarn-site.xml:

<configuration>
    <property>
        <name>yarn.scheduler.minimum-allocation-mb</name>
        <value>128</value>
        <description>Minimum limit of memory to allocate to each container request at the Resource Manager.</description>
    </property>
    <property>
        <name>yarn.scheduler.minimum-allocation-vcores</name>
        <value>1</value>
        <description>The minimum allocation for every container request at the RM, in terms of virtual CPU cores. Requests lower than this won't take effect, and the specified value will get allocated the minimum.</description>
    </property>
    <property>
        <name>yarn.scheduler.maximum-allocation-vcores</name>
        <value>2</value>
        <description>The maximum allocation for every container request at the RM, in terms of virtual CPU cores. Requests higher than this won't take effect, and will get capped to this value.</description>
    </property>
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>2048</value>
        <description>Physical memory, in MB, to be made available to running containers</description>
    </property>
    <property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>4</value>
        <description>Number of CPU cores that can be allocated for containers.</description>
    </property>
</configuration>

Any help is much appreciated.

like image 859
ak. Avatar asked Mar 15 '15 15:03

ak.


1 Answers

Did you try restarting your hadoop's processes or clusters? There might be some works still running.

May be it will be helpful to see the log by following the url of the job or by going through the hadoop url.

Cheers.

like image 187
Spacez Avatar answered Oct 10 '22 07:10

Spacez