Creating a variable zip archive on the fly, estimating file size for content-length

Question

I'm maintaining a site where users can place pictures and other files in a kind of shopping cart. After selecting all the various contents the user wishes to download, he can checkout. Till' now an archive was generated beforehand and the user got an email with the link to the file after the generation finished.

I've changed this now by using web api and push stream to directly generate the archive on the fly. My code is offering either a zip, a zip64 or .tar.gz dynamically, depending on the estimated filesize and operating system. For performance reasons compression ist set to best speed ('none' would make the zip archives incompatible with Mac OS, the gzip library I'm using doesn't offer none).

This is working great so far, however the user is no longer having a progress bar while downloading the file because I'm not setting the content-length. What are the best ways to get around this? I've tried to guess the resulting file size, but either the browsers are canceling the downloads to early or stopping at 99,x% and are waiting for the missing bytes resulting for the difference between the estimated and actual file size.

My first thought was to guess the resulting file size always a little bit to big and filling the rest with zeros?

I've seen many file hosters offering the possibility to select files from a folder and putting them into a zip file and all are having the correct (?) file size with them and a progress bar. Any best practises? Thanks!

Answer 1

This is just some thoughts, maybe you can use them :)

Using Web API/HTTP the normal way to go about is that the response contains the lenght of the file. Since the response is first received after the call has finished, the actual time for generating the file will not show any progress bar in any browser other than a Windows wait cursor.

What you could do is using a two steps approach.

Generating the zip file Create a duplex like channel using SignalR to give feedback on the file generation.

Downloading the zip file After the file is generated you should know the file size, and the browser will show a progress bar while downloading.

Answer 2

It looks that this problem should have been addressed using chunk extensions, but it seems to never got further than a draft.

So I guess you are stuck with either no progress or sending the file size up front.

It seems that generating exact size zip archives is trickier than adding zero padding.

Another option might be to pre-generate the zip file without storing it just to determine the size.

But I am just wondering why not just use tar? It has no compression, so it is easy determine it's final size up front from the size of individual files and it should be also supported by both OSx and Linux. And Windows should be able to handle none compressed zip archives, so a similar trick might work as well.