Append data to a gzip file with Java

Question

I have a log file that may be huge (>4GB) so I am thinking of "gzipping it on the fly", meaning I would like to insert text directly at the end of the gzip file (I want to create my own file appender for logback). Is it possible to append stuff to an existing gzip file? It looks like the only way is to open the existing one with GZIPInputStream, read it and write the data in a new file with GZIPOutputStream and append the new stuff at the end of that new file, then I guess remove the old gzip and rename the new one to the old name...

Am I wrong? Better ideas?

Thanks!

Answer 1

You can look at examples that come in the zlib distribution for how to append to gzip files and how to make a growing gzip file for log entries. You can get the zlib distribution from zlib.net, and look at examples/gzappend.c, examples/gzlog.c, and examples/gzlog.h. They are written in C, and may require zlib capabilities that are not easy to get to using the java.util.zip interface to zlib.

These approaches do not require the recreation of a gzip file.

Answer 2

Pass true to FileOutputStream constructor 2nd argument to get append mode.

File f = new File("out.gz");
OutputStream os = new GZIPOutputStream(new FileOutputStream(f, true)); // true for append
PrintWriter w = new PrintWriter(new OutputStreamWriter(os));
w.println("log message");

On my sample data, this compressed better than the default compression level of gzip. (It was slightly better than level 7, but not as good as level 8).

$ ls -asl out.gz
88 -rw-r--r--  1 jem  staff  41859  5 Jul 07:42 out.gz
$ gunzip out.gz
$ gzip out
$ ls -asl out.gz
88 -rw-r--r--  1 jem  staff  42164  5 Jul 07:42 out.gz