Canvas getImageData returning incorrect data on certain mobile devices

Question

I am working on a canvas video player with some special features based on frames of the video. To overcome the unreliable timing in the video HTML5 tag the videos we are using have a barcode embedded in each frame indicating the current frame number. Using the canvas getImageData method I can grab the pixels and read the barcode to get the frame number. This works great and I have a JSFiddle demonstrating that it works (I couldn't get around CORS in this fiddle to serve the video to the canvas so to see it working you'll have to download the example video locally then upload it via the button. Not ideal but it works).

On certain mobile devices (only Android thus far) this logic breaks. The getImageData returns incorrect values.

It works correctly on my Samsung Galaxy S5 v6.0.1 but fails on a Google Pixel running android v7.1.2. I'll try to collect more data on which devices/OS versions it fails on.

For example, when playing on desktop the first iteration of getImageData returns:

Uint8ClampedArray(64) [3, 2, 3, 255, 255, 255, 255, 255, 246, 245, 247, 255, 243, 242, 243, 255, 241, 239, 241, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255, 242, 240, 242, 255]

which correctly gets computed as framenumber 1.

However on the galaxy, the first iteration returns:

Uint8ClampedArray(64) [255, 242, 217, 255, 255, 234, 209, 255, 41, 1, 1, 255, 254, 235, 210, 255, 255, 234, 209, 255, 50, 4, 0, 255, 254, 240, 215, 255, 255, 248, 224, 255, 255, 249, 225, 255, 255, 251, 232, 255, 255, 252, 233, 255, 255, 252, 233, 255, 255, 253, 234, 255, 255, 255, 237, 255, 255, 255, 237, 255, 28, 1, 1, 255] 

I read that certain devices may being doing additional smoothing so I've been playing around with disabling it in the context via:

this.ctx.mozImageSmoothingEnabled = false;
this.ctx.webkitImageSmoothingEnabled = false;
this.ctx.msImageSmoothingEnabled = false;
this.ctx.imageSmoothingEnabled = false; 

But it didn't help.

Here is the code being used in the JSFiddle.

var frameNumberDiv = document.getElementById('frameNumber');
function load() {
    var canvas = document.getElementById('canvas');
    var ctx = canvas.getContext('2d');
    var video = document.getElementById('video');
    canvas.width = 568;
    canvas.height = 640;
    video.addEventListener('play', function() {
        var that = this; //cache
        (function loop() {
            if (!that.paused && !that.ended) {
                ctx.drawImage(that, 0, 0);
                var pixels = ctx.getImageData(0, 320 - 1, 16, 1).data;
                getFrameNumber(pixels);
                setTimeout(loop, 1000 / 30); // drawing at 30fps
            }
        })();
    }, 0);
}

function getFrameNumber(pixels) {

    let j = 0;
    let thisFrameNumber = 0;
    let str = "Pixels: ";
    for (let i = 0; i < 16; i++) {
        str += pixels[j] + " ";
        thisFrameNumber += getBinary(pixels[j], i);
        j += 4;
    }
    document.getElementById('frameNumber').innerHTML = "FrameNumber: " + thisFrameNumber;
}

function getBinary(pixel, binaryPlace) {
    const binary = [1, 2, 4, 8, 16, 32, 64, 128, 256,
        512, 1024, 2048, 4096, 8192, 16384, 32768
    ];
    if (pixel > 128) return 0;
    if (pixel < 128 && binary[binaryPlace]) {
        return binary[binaryPlace]
    } else {
        return 0;
    }
}

(function localFileVideoPlayer() {
    'use strict';
    var URL = window.URL || window.webkitURL;
    var displayMessage = function(message, isError) {
        var element = document.querySelector('#message');
        element.innerHTML = message;
        element.className = isError ? 'error' : 'info';
    }
    var playSelectedFile = function(event) {
            console.log("Playing");
        var file = this.files[0];
        var type = file.type;
        var videoNode = document.querySelector('video');
        var canPlay = videoNode.canPlayType(type);
        if (canPlay === '') canPlay = 'no';
        var message = 'Can play type "' + type + '": ' + canPlay;
        var isError = canPlay === 'no';
        displayMessage(message, isError);

        if (isError) {
            return;
        }

        var fileURL = URL.createObjectURL(file);
        videoNode.src = fileURL;
        load();
    }
    var inputNode = document.querySelector('input')
    inputNode.addEventListener('change', playSelectedFile, false)
})();

EDIT

• Works on a Nexus 6P running Android v6.0
• Works on a Samsung 6 (Samsung SM G920A) running Android v 5.0.2
• It DOES NOT work on a Samsung Galaxy S7 (SAMSUNG-SM-G935A) running Android v7.0

Could this possibly be an Android 7 issue?

Edit 2

In response to a question in the comments:

videoNode.videoHeight and videoWidth are both 0 on the google pixel for their entire existence but this is the same as on desktop. In both of the devices that don't work that I've encountered the image of each frame is fully painted. I'll attach a screen shot from the google pixel. When paused it consistently reads the same number. In other words it is not jumping around so whatever it is reading is truly on the frame of the video.

EDIT 3: Discovery

I believe that I've made a relevant discovery/realization which I should have seen earlier.

When looking at the output of getImageData on the broken device I was stepping through line by line. What I hadn't (and should have) noticed was that the video element was continuing slightly after it hit my break points / debugger statements. By the time the getImageData method was executed the video had moved past the next frame. So, the scanned barcode was actually for a much later frame than expected.

I added some console log statements and let it run naturally. Looking at the output I can see a much more recognizable pattern.

Here is the first few readings on the google pixel:

Uint8ClampedArray(64) [255, 255, 255, 255, 246, 246, 246, 255, 243, 243, 243, 255, 240, 240, 241, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 241, 241, 242, 255, 2, 2, 2, 255]

Uint8ClampedArray(64) [5, 5, 5, 255, 255, 255, 255, 255, 251, 251, 251, 255, 247, 247, 248, 255, 245, 245, 245, 255, 245, 245, 245, 255, 246, 246, 246, 255, 246, 246, 246, 255, 246, 246, 246, 255, 246, 246, 246, 255, 246, 246, 246, 255, 246, 246, 246, 255, 246, 246, 246, 255, 246, 246, 246, 255, 246, 246, 246, 255, 6, 3, 2, 255]

Uint8ClampedArray(64) [235, 231, 230, 255, 17, 12, 12, 255, 252, 247, 247, 255, 255, 255, 255, 255, 255, 254, 254, 255, 255, 253, 253, 255, 255, 252, 251, 255, 255, 253, 253, 255, 255, 253, 253, 255, 255, 253, 253, 255, 255, 253, 253, 255, 255, 253, 253, 255, 255, 253, 253, 255, 255, 253, 253, 255, 255, 253, 253, 255, 7, 3, 1, 255]

Uint8ClampedArray(64) [26, 15, 14, 255, 4, 0, 0, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 10, 0, 0, 255]

As you may notice, the results seem to be correct however they are shifted one pixel to the left.

I modified the JSFiddle slightly to shift the getImageData read over by a pixel and it gives the exact same response as on the Pixel.

var pixels = ctx.getImageData(1, 320 - 1, 16, 1).data;

Doing -1 seems to have no effect.

So, for some reason these devices are either shifting the entire texture over by a pixel or there is something wrong with the getImageData Method.

EDIT 4

As an experiment I reconfigured my code to use a webGL texture. Same behaviour on desktop/mobile devices. This allowed me to use -1 as the x target using gl.readPixels. I was hoping that by skipping using canvas the entire image would be stored in memory and I could access the pixel data I needed.... Didn't work but here is the data it produced which shows that it is also shifted using purely webGL.

Uint8Array(64) [0, 0, 0, 0, 255, 248, 248, 255, 25, 18, 18, 255, 254, 254, 254, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255]

Uint8Array(64) [0, 0, 0, 0, 255, 254, 247, 255, 255, 244, 236, 255, 48, 18, 10, 255, 254, 246, 238, 255, 255, 247, 239, 255, 255, 247, 239, 255, 255, 248, 241, 255, 255, 251, 243, 255, 255, 251, 243, 255, 255, 251, 243, 255, 255, 251, 243, 255, 255, 251, 243, 255, 255, 251, 243, 255, 255, 250, 240, 255, 255, 250, 240, 255]

Uint8Array(64) [0, 0, 0, 0, 31, 0, 0, 255, 254, 243, 230, 255, 43, 6, 1, 255, 254, 243, 230, 255, 255, 244, 231, 255, 255, 244, 231, 255, 255, 244, 231, 255, 255, 244, 231, 255, 255, 244, 231, 255, 255, 244, 231, 255, 255, 244, 231, 255, 255, 244, 231, 255, 255, 244, 231, 255, 255, 244, 229, 255, 255, 244, 229, 255]

Using:

gl.readPixels(-1, height, 16, 1, gl.RGBA, gl.UNSIGNED_BYTE, pixels);

Edit 5

Ok, I promised I would get more details on which devices were failing/not. I had some online QA testing done using a slightly modified JSFiddle. I modified it slightly to help make it a bit more idiot proof for the general public to work with.

The responses were unfortunately fairly mixed. I was hoping it would be isolated to Android 7 but that doesn't seem to be the case.

I have a CSV on my google drive with the results of this test. Not that these tests are 100% reliable but it seems like it's just some random devices......

Answer 1

I have been going through the same problem and just couldn't get it. I finally have the answer in my case which sounds 99% like your problem as well.

It's the pixel density!!

The pixel densities are different on all the devices you mentioned and the ones that have probably 1 dpr (device pixel ratio) are working correctly, whereas the others are not.

So in my case using p5js I set the pixel density to be 1 and it worked like a charm;

pixelDensity(1);

So set it to be 1 dpr and you are most probably good to go!

I hope this helps some people out there because I spent quite a while on this problem.

Answer 2

Your code is fine so the problem here resides on how some androids render your barcode. The implementation of your barcode is way too small (16x1 pixels) and it's left indented.

Being left indented any anti alias that the device makes on the outer pixels will mess your barcode and give you incorrect results so, when working with video render, you definitely don't want to work on a one pixel safe area.

My suggestion would be to redo the barcodes to a bigger size - let's say 18 pixels height - only use black and white (no grays), center it on the video and the rest of the line paint it green: (in this example it's a barcode for "1")

Then make a GetImageData of the full 320x16 and get rid of everything that has a RGB of 0x255x0 (and approximate) and you have your barcode that would be able to use to get your FrameNumber.

I know you probably would like an answer where it wouldn't be necessary to redo the video but, in this case, the source is the problem.