Receive binary data in web browser from HTTP server in a streaming style

Question

I am looking for a client-side only JavaScript solution that can receive a large binary file from HTTP server to my web client in a streaming style, in which way I can response immediately as soon as I get package of data without having to wait all of them loaded into memory, and I can even discard the data after processing each of them so as to reduce memory footprint.

I searched online, and found it seems not possible via XMLHttpRequest, because of two reasons (quote from this article),

• XHR prevents to access the binary response (with arraybuffer response type) data before the request is complete.
• XHR response is essentially one big buffer that keeps growing linearly as the response data comes in which means it can't get garbage collected.

I wonder whether this can be achieved via websocket by any chance, and is there any good open sources that already solve this issue? I found some that seems relevant, such as Oboe.js and Binary.js, but it either cope with JSON streaming or need server side support.

Answer 1

Use XMLHttpRequest can't fulfill all my requests. But, with some trick I can read the chunk binary data once its arrived. Generally speaking, override the minetype to be 'text/plain; charset=x-user-defined' which will stream the binary data as text, and once one package is ready I can get it and convert it to arrayBuffer.

var xhr = new XMLHttpRequest();
var streamOffset = 0;

xhr.overrideMimeType('text/plain; charset=x-user-defined');
xhr.open("GET", url, true);
xhr.send();
xhr.onreadystatechange = function () {
    var textBuffer = xhr.responseText;
    var arrayBuffer = textToArrayBuffer(textBuffer, streamOffset);
}
function textToArrayBuffer(textBuffer, startOffset) {
    var len = textBuffer.length - startOffset;
    var arrayBuffer = new ArrayBuffer(len);
    var ui8a = new Uint8Array(arrayBuffer, 0);
    for (var i = 0, j = startOffset; i < len; i++, j++)
        ui8a[i] = (textBuffer.charCodeAt(j) & 0xff);
    return arrayBuffer;
}

Though, in this way I can get binary data in a streaming way, it can't be throw away after processing each chunk until the request is complete. Anyway, this give me the chance to handle the binary data as soon as it arrives.