.NET DeflateStream vs linux zlib difference

Question

I need cross-platform compression/decompression between Windows and Ubuntu. As I understand, starting .NET 4.5, class DeflateStream uses zlib as the compression libraries. I have written two small test programs to compress data, one in C# running on Windows and the other in 'C' running on Ubuntu. The .NET platform is 4.5.2.

C# code is using CompressionLevel.Optimal

C code is using Z_BEST_COMPRESSION

Here are the results:

Input bytes: {9, 12, 13}
C# output: {227, 228, 225, 5, 0};
C output:  {120, 218, 227, 228, 225, 5, 0, 0, 67, 0, 35}

As you can see, the length of compressed data in C# is 5 bytes where as in C is 11 bytes. It seems 'C' zlib is adding 2 extra bytes in header and 4 extra bytes in footer.

If needed, I can share the code. However, it is taken from standard examples that you see on the net and there is nothing special about the code.

Am I missing something? Is there a way to fix it? If the header and the footer always stay the same, perhaps I can always add the additional bytes. Regards.

Answer 1

First, you need to understand that there are three possible formats that zlib can generate. They are raw deflate (RFC 1951), a zlib stream which is raw deflate wrapped in a zlib header and trailer (RFC 1950), and a gzip stream which is raw deflate wrapped in a gzip header and trailer (RFC 1952). Your C# code is generating a raw deflate stream, whereas your C code is generating a zlib stream.

You have not shown your code, but you can easily use zlib (in your C code) to produce a raw deflate stream, as DeflateStream does. Unfortunately, there is no class in NET 4.5 to produce a zlib stream (in your C# code). However, you can easily create your own zlib header and trailer to wrap the raw deflate stream with. (See the RFC.)

However I would highly recommend that you not use the NET 4.5 zlib interface routines at all. Use DotNetZip instead, which provides an interface to the full functionality of zlib, and more, and most importantly does not have bugs like in NET 4.5 that Microsoft has said that they will not fix!

Answer 2

Turns out this is a very old incompatibility problem between DeflateStream and zlib as mentioned here: https://tlzprgmr.wordpress.com/2010/03/17/net-deflatestreamzlib-compatibility/

Essentially, DeflateStream does not add the required header or footer to the compressed data. Regards.