Receiving image through websocket

Question

I am using websockify to display images from a python server to a HTML5 canvas.

I think that I have manage to successfully send images from my python server but I am not able to display the images to my canvas.

I think the problem has to do with the number of bytes that I am trying to display on canvas and I believe that I am not waiting until the whole image is received and then displaying the image to the canvas.

Until now I have:

The on message function. When I sent an image I get 12 MESSAGERECEIVED in console

  ws.on('message', function () {
    //console.log("MESSAGERECEIVED!")
            msg(ws.rQshiftStr());
  });

The msg function where I receive the string and I am trying to display it on canvas. I invoking the method 12 times for each picture. The format of the sting is 'xÙõKþ°pãüCY :

function msg(str) {
        //console.log(str);
        console.log("RELOAD");

        var ctx = cv.getContext('2d');
        var img = new Image();
        //console.log(str);
        img.src = "data:image/png;base64," + str;
        img.onload = function () {
            ctx.drawImage(img,0,0);
        }
    }

Any suggestions on how to fix this?

Answer 1

The focus of websockify+websock.js is to transparently support streaming binary data (more on that below). The data you get off the receive queue is already base64 decoded. However, the data URI scheme is expecting a base64 encoded string so you need to encode the image data to base64. You can use the builtin window.btoa() to base64 encode a binary coded string:

img.src = "data:image/png;base64," + window.btoa(str);

Or, for greater efficiency you can use the Base64 module from include/base64.js but you will need to pass it an array of bytes as your would get from rQshiftBytes:

msg(ws.rQshiftBytes());

img.src = "data:image/png;base64," + Base64.encode(data);  // str -> data

Regarding the use of base64 in websockify:

Websockify uses base64 to encode the data to support browsers which don't support binary data directly. In addition to such popular Hixie only browsers such as iOS Safari and desktop Safari, some browsers versions in the wild use HyBi but are missing binary support. And unfortunately, in the case of Chrome, they also had a WebIDL bug around that same time which prevents detecting binary support before making a connection.

Also, the main option for using WebSockets on Opera, firefox 3.6-5, IE 8 and 9 is web-socket-js. web-socket-js supports HyBi but does not have binary support and probably won't because most of the older browsers that it targets don't support native binary types (Blob and Typed Arrays).

The market share of browsers which support HyBi and binary data is currently pretty low. However, in the future, Websockify will detect (either by object detection or browser version detection) whether the browser supports binary data and use that support directly without the need for base64 encode/decode.

The latency (and CPU) overhead of base64 encode/decode is pretty low and usually washed out by network latencies anyhow. The bandwidth overhead of base64 encoded data is about 25% (i.e. raw data becomes 33% larger), but it does give you binary data over WebSockets on basically all browsers in the wild (with the web-socket-js polyfill/shim which is used transparently by websockify if needed).