Save client generated data as file in JavaScript in chunks

Question

I'm developing a FileShare application with webRTC. I want to implement the client in JavaScript/HTML. The code should be run on the clients browser. I need to save them when downloaded via webRTC. The files can be quite big and I can't completely donwload them and save them in a array or blob before saving them to disk as a file.

Is there any API that allows me to save the file in chunks as I recieve them?

I have found so far Downloadify, FileSave.js and html5 FileWriterApi so far. While the first two are not chunked and require me to first download the complete file to memory before saving, the FileWriterAPI is not available on most browsers.

Answer 1

As @jordan-gray suggested, saving the chunks in blobs and joining them to a larger blob might be a solution if:

• Persistence of chunks is not needed (i.e. closing the browser will delete all chunks)
• The file is persisted only by the user saving it to his own filesystem. The web application will not have access to the file once it is closed, unless the user gave access to the saved file again.
• Possibly, if the file sizes are not too big (you'll have to benchmark to find that out). Chrome was behaving quite nice for me for chunks totaling at 1GB.

I've created a simple test for using blobs as chunks. You can play around with the different size and chunk numbers parameters:

var chunkSize = 500000;
var totalChunks = 200;
var currentChunk = 0;
var mime = 'application/octet-binary';
var waitBetweenChunks = 50;

var finalBlob = null;
var chunkBlobs =[];

function addChunk() {
    var typedArray = new Int8Array(chunkSize);
    chunkBlobs[currentChunk] = new Blob([typedArray], {type: mime});
    console.log('added chunk', currentChunk);
    currentChunk++;
    if (currentChunk == totalChunks) {
        console.log('all chunks completed');
        finalBlob = new Blob(chunkBlobs, {type: mime});
        document.getElementById('completedFileLink').href = URL.createObjectURL(finalBlob);
    } else {
        window.setTimeout(addChunk, waitBetweenChunks);
    }
}
addChunk();

If you do need that persistence, the W3C File System API should support what you need. You can use it to write the chunks to separate files, and then when all the chunks are completed you can read them all and append them to a single file, and remove the chunks.

Note that it works by assigning a sandboxed filesystem for your application (for a given quota), and the files are only accessible to that application. If the files are meant to use outside of the web application, you might need the function for the use to save the file from the application filesystem to his "normal" filesystem. You can do something like that using the createObjectURL() method.

You are right about current state of browser support. A Filesystem API polyfill is available, which is based on IndexedDB (which is more widely supported) as a filesystem emulation backend. I did not test the polyfill on large files. You might run into size limits or performance limitations.

Answer 2

Did you check https://github.com/Peer5/Sharefest out ? It should cover your requirements