Are there any free compression utilities that zip in real time like Xceed Real-Time Zip?

Question

What I need to do is compress 64KB chunks of a file as they're coming in from an external service and then stream that chunk of compressed data to the browser all in real time. Xceed Real-Time is pretty awesome in how you can set the header of the file and then piece together the multiple 64KB chunks into a whole zipped file on the client. This works great and you can easily handle multiple files by putting it in a loop.

My question is this: Is there a free alternative compression utility that can match the sophistication of Xceed? I need something that can compress and stream parts of a file to the browser. Pretty much all of the free alternatives I've seen require having access to the entire file(s) before compression can take place. Xceed is great, but it's expensive. Just wondering if there's a free alternative out there that can accomplish this. Thanks!

Answer 1

What you are looking for is stream compression approach. Good news is you have quite a lot of options. Please also note you have to think about memory because some solutions (dictionary-based) might require a lot of memory, so test and tune.

In order to keep my answer short I would suggest to give LZ4 a try.

LZ4 - http://code.google.com/p/lz4/

LZ4 is a very fast lossless compression algorithm, providing compression speed at 300 MB/s per core, scalable with multi-cores CPU. It also features an extremely fast decoder, with speed in GB/s per core, typically reaching RAM speed limits on multi-core systems.

Benchmark:

Name            Ratio  C.speed D.speed
                        MB/s    MB/s
LZ4 (r97)       2.084    410    1810
LZO 2.06        2.106    409     600
QuickLZ 1.5.1b6 2.237    373     420
Snappy 1.1.0    2.091    323    1070
LZF             2.077    270     570
zlib 1.2.8 -1   2.730     65     280
LZ4 HC (r97)    2.720     25    2040
zlib 1.2.8 -6   3.099     21     300

There are a lot of implementations available, please check: http://code.google.com/p/lz4/

• C# - https://github.com/stangelandcl/LZ4Sharp

LZO - http://lzo-net.sourceforge.net/

LZO.Net brings the power of Markus "FXJ" Oberhumer's great LZO compression library (V1.08) to .Net. It wraps the access to the native DLL with a small C# class maintaining the raw speed of the ANSI-C library.

Snappy - https://code.google.com/p/snappy/

Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression.

Two C# native implementations are available:

• SnappySharp - https://github.com/Kintaro/SnappySharp
• Snappy.Sharp - https://github.com/jeffesp/Snappy.Sharp

QuickLZ - http://www.quicklz.com/

QuickLZ is the world's fastest compression library, reaching 308 Mbyte/s per core. It can be used under a commercial license if such has been acquired or under GPL 1, 2 or 3 where anything released into public must be open source.

Native implementations - QuickLZ C#

So far, only a subset of the library has been ported, 
namely the setting:

QLZ_COMPRESSION_LEVEL = 1 or 3 
QLZ_STREAMING_BUFFER = 0 
QLZ_MEMORY_SAFE = 0