Ruby zlib deflate massive data

Question

I'm trying to use Zlib::Deflate.deflate on a massive file (4 gigs). There are obvious problems with doing that, the first of which being that I can't load the entire file into memory all at once. Zlib::GzipWriter would work, since it works with streams, but it's not zlib compression. Any ideas?

Answer 1

You could try instantiating a Zlib::Deflate stream and feeding it data from your big file piecemeal. Zlib::Deflate::deflate purports to do that sort of thing behind the scenes.

It would look something like this:

z = Zlib::Deflate.new

File.open "big_uncompressed_file" do |f|
  File.open "big_compressed_file", "w" do |w|
    f.each do |str|
      w << z.deflate str, Zlib::SYNC_FLUSH
    end
  end
end
z.finish
z.close

ruby zlib docs

notes on zlib flush flags