How to efficiently deal with a large amount of HTML5 canvas pixel data over websockets [duplicate]

Question

Possible Duplicate:
Receiving image through websocket

Using

imageData = context.getImageData(0, 0, width, height);
JSON.stringify(imageData.data);

I grab the pixel data, convert it to a string, and then send it over the wire via websockets. However, this string can be pretty large, depending on the size of the canvas object. I tried using the compression technique found here: JavaScript implementation of Gzip but socket.io throws the error Websocket message contains invalid character(s). Is there an effective way to compress this data so that it can be sent over websockets?

Answer 1

There are several ways I would recommend depending on which axis of efficiency you are wanting (bandwidth vs CPU efficiency).

Option 1: You can use the canvas toDataURL method. This returns a base64 encoded image of the canvas image data. It will be compressed using the image format you specify (or PNG for the default) and it will be pre-encoded to base64 for sending over WebSocket.

canvas = document.getElementById("mycanvas");
b64png = canvas.toDataURL();

ws.send(b64png);

Option 2: If you can tolerate lossy compression then you can ask for the image as a base64 encoded JPEG from the toDataURL method:

canvas = document.getElementById("mycanvas");
b64jpeg = canvas.toDataURL("image/jpeg");

ws.send(b64jpeg);

Option 3: If you are using a browser that supports binary WebSocket data (Chrome, Firefox, IE 10) then you can just send the canvas arraybuffer directly over WebSocket

canvas = document.getElementById("mycanvas");
ctx = canvas.getContext('2d');
imgdata = ctx.getImageData(0,0, width, height).data; // This is a Uint8ClampedArray
ws.send(imgdata.buffer); // Send the ArrayBuffer from the Uint8ClampedArray

Option 3 will likely be the least efficient in terms of bandwidth, but the most efficient in terms of processing power on the client and server side because the image data is sent raw with little pre/post processing required.

The most bandwidth efficient option will likely be #2 but you will lose some image quality during conversion of the image data to JPEG format. You could even go further and base64 decode the data into an arraybuffer or blob and send that via binary WebSocket so that you don't get the 33% base64 bandwidth overhead, but this adds even more CPU overhead.

If you want efficient bandwidth without losing any image quality then option #2 is your best bet.

Some notes/caveats:

The toDataURL prefixes the base64 data something like this:

"data:image/png;base64,iVBORw0KGgoAAAA..."

One nice thing about the data URL format is that you can take the whole thing and paste it into your browsers address bar and the browser will render the image.

See the MDN Canvas page for more info about toDataURL.

Answer 2

The most bandwidth efficient way is to to send photo like data is JPEG encoded binary as blob

You can get <canvas> data as binary JPEG blob:

https://github.com/miohtama/Krusovice/blob/master/src/tools/resizer.js#L51

(For non-photo like content you can also get PNG blob)

Blob is always raw binary, no UTF-8 or base64 crap involved.

WebSocket.send() supports blobs as input:

https://developer.mozilla.org/en/WebSockets/WebSockets_reference/WebSocket

HTTP Blob sending:

https://developer.mozilla.org/en/DOM/XMLHttpRequest/Sending_and_Receiving_Binary_Data

Your mileage with different browsers may vary.