How to read a .gz file line-by-line in C++?

Question

I have 3 terabyte .gz file and want to read its uncompressed content line-by-line in a C++ program. As the file is quite huge, I want to avoid loading it completely in memory.

Can anyone post a simple example of doing it?

Answer 1

You most probably will have to use ZLib's deflate, example is available from their site

Alternatively you may have a look at BOOST C++ wrapper

The example from BOOST page (decompresses data from a file and writes it to standard output)

#include <fstream> #include <iostream> #include <boost/iostreams/filtering_streambuf.hpp> #include <boost/iostreams/copy.hpp> #include <boost/iostreams/filter/zlib.hpp>  int main()  {     using namespace std;      ifstream file("hello.z", ios_base::in | ios_base::binary);     filtering_streambuf<input> in;     in.push(zlib_decompressor());     in.push(file);     boost::iostreams::copy(in, cout); } 

Answer 2

For something that is going to be used regularly, you probably want to use one of the previous suggestions. Alternatively, you can do

gzcat file.gz | yourprogram 

and have yourprogram read from cin. This will decompress parts of the file in memory as it is needed, and send the uncompressed output to yourprogram.