Menu
We've realized a bit too late that archiving our files in GZip format for Hadoop processing isn't such a great idea. GZip isn't splittable, and for reference, here are the problems which I won't repeat:
My question is: is BZip2 the best archival compression that will allow a single archive file to be processed in parallel by Hadoop? Gzip is definitely not, and from my reading LZO has some problems.
796
BZIP2 is splittable in hadoop - it provides very good compression ratio but from CPU time and performances is not providing optimal results, as compression is very CPU consuming. LZO is splittable in hadoop - leveraging hadoop-lzo you have splittable compressed LZO files. You need to have external .
bzip2 has notably better compression ratio than gzip, which has to be the reason for the popularity of bzip2; it is slower than gzip especially in decompression and uses more memory. However the memory requirements of bzip2 should be nowadays no problem even on older hardware.
Moreover, like those programs, the compression is lossless, meaning that no data is lost during compression and thus the original files can be exactly regenerated. The only disadvantage of bzip2 is that it is somewhat slower than gzip and zip.
I don't consider the other answer correct, bzip2 according to this:
http://comphadoop.weebly.com/
is splittable. LZO is too if indexed.
So the answer is yes, if you want to use more mappers than you have files, then you'll want to use bzip2.
To do this, you could write a simple MR job to read the data then just write it out again, you then need to ensure you set mapred.output.compression.codec to org.apache.hadoop.io.compress.BZip2Codec
45
BZIP2 is splittable in hadoop - it provides very good compression ratio but from CPU time and performances is not providing optimal results, as compression is very CPU consuming.
LZO is splittable in hadoop - leveraging hadoop-lzo you have splittable compressed LZO files. You need to have external .lzo.index files to be able to process in parallel. The library provides all means of generating these indexes in local or distributed manner.
LZ4 is splittable in hadoop - leveraging hadoop-4mc you have splittable compressed 4mc files. You don't need any external indexing, and you can generate archives with provided command line tool or by Java/C code, inside/outside hadoop. 4mc makes available on hadoop LZ4 at any level of speed/compression-ratio: from fast mode reaching 500 MB/s compression speed up to high/ultra modes providing increased compression ratio, almost comparable with GZIP one.
75
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
