Only decompress a specific bzip2 block

Question

Say I have a bzip2 file (over 5GB), and I want to decompress only block #x, because there is where my data is (block is different every time). How would I do this?

I thought about making an index of where all the blocks are, then cut the block I need from the file and apply bzip2recover to it.

I also thought about compressing say 1MB at a time, then appending this to a file (and recording the location), and simply grabbing the file when I need it, but I'd rather keep the original bzip2 file intact.

My preferred language is Ruby, but any language's solution is fine by me (as long as I understand the principle).

Answer 1

There is a http://bitbucket.org/james_taylor/seek-bzip2

Grab the source, compile it.

Run with

./seek-bzip2  32 < bzip_compressed.bz2 

to test.

the only param is bit displacement of wondered block header. You can get it with finding a "31 41 59 26 53 59 " hex string in the binary file. THIS WAS INCORRECT. Block start may be not aligned to byte boundary, so you should search for every possible bit shifts of "31 41 59 26 53 59" hex string, as it is done in bzip2recover - http://www.bzip.org/1.0.3/html/recovering.html

32 is bit size of "BZh1" header where 1 can be any digit from "1" to "9" (in classic bzip2) - it is a (uncompressed) block size in hundreds of kb (not exact).

Answer 2

It's true that bzip-table is almost as slow as decompressing but of course you only have to do it once and you can store the output in some fashion to use as an index. This is perfect for what I need but may not be what everybody needs.

I did need a little help getting it to compile on Windows though.