Is there a library for creating zip files (the zip file format not gzip or any other compression format) on-the-fly (so I can start sending the file while it is compressing) for very large files (4 Gb and above).
The compression ratio does not matter much (mostly media files).
The library has to have a c-interface and work on Debian and OSX.
Limitations. It's not possible to completely stream-write ZIP files. Small bits of metadata for each member file, such as its name, must be placed at the end of the ZIP. In order to do this, stream-zip buffers this metadata in memory until it can be output.
The ZIP file format uses lossless compression algorithms to do exactly that. It allows you to express the same information in a more efficient way by removing the redundant data from the file. This also means it is faster to send a ZIP file.
For the best compression rate, choose 7z. Compression level — the compression time increases with the compression level. The presets range from Store (fastest compression) to Ultra (slowest compression time with the most space saved).
What is 7-Zip? 7-Zip is a file archive with the highest compression ratio. It works on the 7z format, which is the successor to the zip format. This format enables it to achieve more than a 1350 percent compression ratio compared to the zip format.
libarchive supports any format you want, on the fly and even in-memory files.
zlib supports compressing by chunks. you should be able to start sending a small chunk right after compressing it, while the library is still compressing the next chunk. (see this example)
(unfortunately, the file table is stored at the end of the zip file, so the file will be unusable until it is complete on the receiver side)
While this question is old and already answered I will note a new potential solution for those that find this.
I needed something very similar, a portable and very small library that created ZIP archives in a streaming fashion in C. Not finding anything that fit the bill I created one that uses zlib, available here:
https://github.com/CTrabant/fdzipstream
That code only depends on zlib and essentially provides a simple interface to creating ZIP archives. Most importantly (for me) the output can be streamed to a pipe, socket, whatever as the output stream does not need to be seek-able. The code is very small, a single source file and a header file. Works on OSX and Linux and probably elsewhere. Hope it helps someone beyond just me...
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With