Menu
Can anyone explain how the RecordReader actually works? How are the methods nextkeyvalue(), getCurrentkey() and getprogress() work after the program starts executing?
914
RecordReader , typically, converts the byte-oriented view of the input, provided by the InputSplit , and presents a record-oriented view for the Mapper and Reducer tasks for processing. It thus assumes the responsibility of processing record boundaries and presenting the tasks with keys and values.
In MapReduce, RecordReader load data from its source and it converts the data into key-value pairs suitable for reading by the mapper. RecordReader communicates with the inputsplit until it does not read the complete file. The MapReduce framework defines RecordReader instance by the InputFormat.
Hadoop InputFormat describes the input-specification for execution of the Map-Reduce job. InputFormat describes how to split up and read input files. In MapReduce job execution, InputFormat is the first step. It is also responsible for creating the input splits and dividing them into records.
RecordWriter is the class which handles the job of taking an individual key-value pair i.e output from reducer and writing it to the location prepared by the OutputFormat. RecordWriter implements: 'write' and 'close'. The 'write' function takes key-values from the MapReduce job and writes the bytes to HDFS.
(new API): The default Mapper class has a run method which looks like this:
public void run(Context context) throws IOException, InterruptedException {
setup(context);
while (context.nextKeyValue()) {
map(context.getCurrentKey(), context.getCurrentValue(), context);
}
cleanup(context);
}
The Context.nextKeyValue(), Context.getCurrentKey() and Context.getCurrentValue() methods are wrappers for the RecordReader methods. See the source file src/mapred/org/apache/hadoop/mapreduce/MapContext.java.
So this loop executes and calls your Mapper implementation's map(K, V, Context) method.
Specifically, what else would you like to know?
125
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
