I have the following code and I didn't understand why get()
method has been used in the highlighted line. If I remove that get()
method it throws me an error.
What I can take out from it is: get()
method returns the int value of the IntWritable. Correct me if I am wrong.
public void reduce(IntWritable key, Iterator<IntWritable> values, OutputCollector<IntWritable, IntWritable> output, Reporter reporter) throws IOException {
int sum = 0;
while (values.hasNext()) {
sum += values.next().get(); //error when removing the get()
}
}
Your understanding is correct. RamPrasad G's answer is also correct (+1), but, just to make it clearer:
sum
is int. values
is an Iterator
to IntWritable
elements, so values.next()
is an IntWritable
. Now, IntWritable
, a data type of hadoop, is not equivalent to the primitive type int
of java, even if they are used for the same thing: to store integer values.
Thus, you cannot add an IntWritable
to an int
(which is what you do if you remove the get()
method). They are not of the same type. You have to convert the IntWritable
to an int
, which is done by the get()
method.
In order to handle the Objects in Hadoop way, Hadoop uses Writable classes. For example, Hadoop uses Text instead of java's String. similarly, The IntWritable class in Hadoop is similar to a java int, however, IntWritable implements interfaces like Comparable, Writable and WritableComparable.
These interfaces are all necessary for MapReduce; the Comparable
interface is used for comparing when the reducer sorts the keys, and Writable
can write the result to the local disk. It does not use the java Serializable
because java Serializable
is too big or too heavy for hadoop, Writable
can serializable the hadoop Object in a very light way.
Writable interface is described as
A serializable object which implements a simple, efficient, serialization protocol, based on
DataInput
andDataOutput
Your values.next()
is IntWritable class you have to use get()
method to get primitive type of it.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With