Menu
I am currently running a job I fixed the number of map task to 20 but and getting a higher number. I also set the reduce task to zero but I am still getting a number other than zero. The total time for the MapReduce job to complete is also not display. Can someone tell me what I am doing wrong. I am using this command
hadoop jar Test_Parallel_for.jar Test_Parallel_for Matrix/test4.txt Result 3 \ -D mapred.map.tasks = 20 \ -D mapred.reduce.tasks =0 Output:
11/07/30 19:48:56 INFO mapred.JobClient: Job complete: job_201107291018_0164 11/07/30 19:48:56 INFO mapred.JobClient: Counters: 18 11/07/30 19:48:56 INFO mapred.JobClient: Job Counters 11/07/30 19:48:56 INFO mapred.JobClient: Launched reduce tasks=13 11/07/30 19:48:56 INFO mapred.JobClient: Rack-local map tasks=12 11/07/30 19:48:56 INFO mapred.JobClient: Launched map tasks=24 11/07/30 19:48:56 INFO mapred.JobClient: Data-local map tasks=12 11/07/30 19:48:56 INFO mapred.JobClient: FileSystemCounters 11/07/30 19:48:56 INFO mapred.JobClient: FILE_BYTES_READ=4020792636 11/07/30 19:48:56 INFO mapred.JobClient: HDFS_BYTES_READ=1556534680 11/07/30 19:48:56 INFO mapred.JobClient: FILE_BYTES_WRITTEN=6026699058 11/07/30 19:48:56 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=1928893942 11/07/30 19:48:56 INFO mapred.JobClient: Map-Reduce Framework 11/07/30 19:48:56 INFO mapred.JobClient: Reduce input groups=40000000 11/07/30 19:48:56 INFO mapred.JobClient: Combine output records=0 11/07/30 19:48:56 INFO mapred.JobClient: Map input records=40000000 11/07/30 19:48:56 INFO mapred.JobClient: Reduce shuffle bytes=1974162269 11/07/30 19:48:56 INFO mapred.JobClient: Reduce output records=40000000 11/07/30 19:48:56 INFO mapred.JobClient: Spilled Records=120000000 11/07/30 19:48:56 INFO mapred.JobClient: Map output bytes=1928893942 11/07/30 19:48:56 INFO mapred.JobClient: Combine input records=0 11/07/30 19:48:56 INFO mapred.JobClient: Map output records=40000000 11/07/30 19:48:56 INFO mapred.JobClient: Reduce input records=40000000 [hcrc1425n30]s0907855: 
683
The number of map tasks for a given job is driven by the number of input split. For each input split or HDFS blocks a map task is created. So, over the lifetime of a map-reduce job the number of map tasks is equal to the number of input splits.
Every job consists of two key components: mapping task and reducing task. The map task plays the role of splitting jobs into job-parts and mapping intermediate data. The reduce task plays the role of shuffling and reducing intermediate data into smaller units. The job tracker acts as a master.
The mapper processes the data and creates several small chunks of data. Reduce stage − This stage is the combination of the Shuffle stage and the Reduce stage. The Reducer's job is to process the data that comes from the mapper. After processing, it produces a new set of output, which will be stored in the HDFS.
The number of map tasks for a given job is driven by the number of input splits and not by the mapred.map.tasks parameter. For each input split a map task is spawned. So, over the lifetime of a mapreduce job the number of map tasks is equal to the number of input splits. mapred.map.tasks is just a hint to the InputFormat for the number of maps.
In your example Hadoop has determined there are 24 input splits and will spawn 24 map tasks in total. But, you can control how many map tasks can be executed in parallel by each of the task tracker.
Also, removing a space after -D might solve the problem for reduce.
For more information on the number of map and reduce tasks, please look at the below url
https://cwiki.apache.org/confluence/display/HADOOP2/HowManyMapsAndReduces
62
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
