Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How do I transparently compress/decompress a file as a program writes to/reads from it?

I have a program that reads and writes very large text files. However, because of the format of these files (they are ASCII representations of what should have been binary data), these files are actually very easily compressed. For example, some of these files are over 10GB in size, but gzip achieves 95% compression.

I can't modify the program but disk space is precious, so I need to set up a way that it can read and write these files while they're being transparently compressed and decompressed.

The program can only read and write files, so as far as I understand, I need to set up a named pipe for both input and output. Some people are suggesting a compressed filesystem instead, which seems like it would work, too. How do I make either work?

Technical information: I'm on a modern Linux. The program reads a separate input and output file. It reads through the input file in order, though twice. It writes the output file in order.

like image 226
A. Rex Avatar asked Apr 16 '09 07:04

A. Rex


People also ask

How do I decompress a program?

Right-click the file you want to zip, and then select Send to > Compressed (zipped) folder. Open File Explorer and find the zipped folder. To unzip the entire folder, right-click to select Extract All, and then follow the instructions. To unzip a single file or folder, double-click the zipped folder to open it.

What does it mean to decompress a file?

Decompression is the process of restoring compressed data to its original form. Data decompression is required in almost all cases of compressed data, including lossy and lossless compression.

Which command is use for compress and decompress the file?

The bzip2 command is used to compress and decompress the files i.e. it helps in binding the files into a single file which takes less storage space than the original file used to take.


1 Answers

Check out zlibc: http://zlibc.linux.lu/.

Also, if FUSE is an option (i.e. the kernel is not too old), consider: compFUSEd http://www.biggerbytes.be/

like image 169
EFraim Avatar answered Nov 11 '22 06:11

EFraim