Merging multiple sequence files into one sequencefile within Hadoop

Question

How to merge multiple sequence files into one sequence file within Hadoop Thanks.

Answer 1

If you want to merge multiple files into single file then here is two ans :

Native language

---

getmerge

Usage: hadoop fs -getmerge <src> <localdst>

Takes a source directory and a destination file as input and concatenates files in src into the destination local file. Optionally addnl can be set to enable adding a newline character at the end of each file.

---

---

Java API

---

org.apache.hadoop.fs.FileUtil.copyMerge(FileSystem srcFS, Path srcDir, FileSystem dstFS, Path dstFile, boolean deleteSource, Configuration conf, String addString);

Copy all files in a directory to one output file (merge)

Copy to hdfs

put

Usage: hadoop dfs -put <localsrc> ... <dst>

Copy single src, or multiple srcs from local file system to the destination filesystem. Also reads input from stdin and writes to destination filesystem.

copyFromLocal

Usage: hadoop dfs -copyFromLocal <localsrc> URI

Similar to put command, except that the source is restricted to a local file reference.

Answer 2

Have you considered forqlift? I wrote it to handle certain SequenceFile chores, including SequenceFile merges.

In your case, you could run:

forqlift seq2seq --file new_combined_file.seq  \
    original_file1.seq  original_file2.seq original_file3.seq ...

Granted, forqlift's seq2seq tool is marked "experimental" ... but it's worked well on my (admittedly limited) internal testing.