Merging multiple files into one within Hadoop

Question

I get multiple small files into my input directory which I want to merge into a single file without using the local file system or writing mapreds. Is there a way I could do it using hadoof fs commands or Pig?

Thanks!

Answer 1

In order to keep everything on the grid use hadoop streaming with a single reducer and cat as the mapper and reducer (basically a noop) - add compression using MR flags.

hadoop jar \     $HADOOP_PREFIX/share/hadoop/tools/lib/hadoop-streaming.jar \<br>     -Dmapred.reduce.tasks=1 \     -Dmapred.job.queue.name=$QUEUE \     -input "$INPUT" \     -output "$OUTPUT" \     -mapper cat \     -reducer cat 

If you want compression add
-Dmapred.output.compress=true \ -Dmapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec

Answer 2

hadoop fs -getmerge <dir_of_input_files> <mergedsinglefile>