Advantages of using NullWritable in Hadoop

Question

What are the advantages of using NullWritable for null keys/values over using null texts (i.e. new Text(null)). I see the following from the «Hadoop: The Definitive Guide» book.

NullWritable is a special type of Writable, as it has a zero-length serialization. No bytes are written to, or read from, the stream. It is used as a placeholder; for example, in MapReduce, a key or a value can be declared as a NullWritable when you don’t need to use that position—it effectively stores a constant empty value. NullWritable can also be useful as a key in SequenceFile when you want to store a list of values, as opposed to key-value pairs. It is an immutable singleton: the instance can be retrieved by calling NullWritable.get()

I do not clearly understand how the output is written out using NullWritable? Will there be a single constant value in the beginning output file indicating that the keys or values of this file are null, so that the MapReduce framework can ignore reading the null keys/values (whichever is null)? Also, how actually are null texts serialized?

Thanks,

Venkat

Answer 1

The key/value types must be given at runtime, so anything writing or reading NullWritables will know ahead of time that it will be dealing with that type; there is no marker or anything in the file. And technically the NullWritables are "read", it's just that "reading" a NullWritable is actually a no-op. You can see for yourself that there's nothing at all written or read:

NullWritable nw = NullWritable.get(); ByteArrayOutputStream out = new ByteArrayOutputStream(); nw.write(new DataOutputStream(out)); System.out.println(Arrays.toString(out.toByteArray())); // prints "[]"  ByteArrayInputStream in = new ByteArrayInputStream(new byte[0]); nw.readFields(new DataInputStream(in)); // works just fine 

And as for your question about new Text(null), again, you can try it out:

Text text = new Text((String)null); ByteArrayOutputStream out = new ByteArrayOutputStream(); text.write(new DataOutputStream(out)); // throws NullPointerException System.out.println(Arrays.toString(out.toByteArray())); 

Text will not work at all with a null String.