Get frame numbers in HTML5 Video

Question

I am trying to capture each frame number of the video however it looks like there is no way of achieving it. So I started my own clock to match the frame numbers of the video but they never match and the difference keeps increasing as the video progress.

Please have a look at my bin. http://jsbin.com/dopuvo/4/edit

I have added the frame number to each frame of the video from Adobe After Effect so I have more accurate information of the difference. The Video is running at 29.97fps and the requestAnimationFrame is also set to increase the number at the same rate, however I am not sure where this difference is coming from.

Sometimes they match and sometimes they don't. I also tried doing it offline but I get the same results. Any help.

Answer 1

I found something on github for this. https://github.com/allensarkisyan/VideoFrame

I have implemented it in this fiddle: https://jsfiddle.net/k0y8tp2v/

var currentFrame = $('#currentFrame');
var video = VideoFrame({
    id : 'video',
    frameRate: 25,
    callback : function(frame) {
        currentFrame.html(frame);
    }
});

$('#play-pause').click(function(){
    if(video.video.paused){
        video.video.play();
        video.listen('frame');
        $(this).html('Pause');
    }else{
        video.video.pause();
        video.stopListen();
        $(this).html('Play');
    }
});

EDIT: updated fiddle to new video so it works again.

EDIT: As pointed out, the video is 25fps, so I updated it, and while I was there removed reliance on jQuery.
Non jQuery version:
https://jsfiddle.net/k0y8tp2v/1/

var currentFrame = document.getElementById('currentFrame');
var video = VideoFrame({
    id : 'video',
    frameRate: 25,
    callback : function(frame) {
        currentFrame.innerHTML = frame ;
    }
});

document.getElementById('play-pause').addEventListener('click', function(e){
    if(video.video.paused){
        video.video.play();
        video.listen('frame');
        e.target.innerHTML = 'Pause';
    }else{
        video.video.pause();
        video.stopListen();
        e.target.innerHTML = 'Play';
    }
});

Answer 2

The problem is that setTimeout is not really predictable, so you can't be sure that exactly one new frame has been displayed every time your function runs. You need to check the currentTime of the video every time you update your frame display and multiply that by the frame rate.

Here's a working example: http://jsbin.com/xarekice/1/edit It's off by one frame, but it looks like you may have two frames at the beginning marked "000000".

A few things about the video element that you may want to be aware of:

1. As you seem to have discovered, there's no reliable way to determine the frame rate, so you have to discover it elsewhere and hard-code it. There are some interesting things going on with video metrics, but they're non-standard, not widely supported and, in my experience, completely ineffective at determining the actual frame rate.

2. The currentTime is not always exactly representative of what's on the screen. Sometimes it's ahead a little bit, especially when seeking (which is why in my JSBin, I don't update the frame while you're seeking).

I believe currentTime updates on a separate thread from the actual video draw, so it kind of works like it's own clock that just keeps going. It's where the video wants to be, not necessarily where it is. So you can get close, but you need to round the results of the frame calculation, and once in a while, you may be off by one frame.

Answer 3

Starting in M83, Chromium has a requestVideoFrameCallback() API, which might solve your issue. You can use the mediaTime to get a consistent timestamp, as outlined in this Github issue. From there, you could try something like this:

var frameCounter = (time, metadata) => {
   let count = metadata.mediaTime * frameRate;
   console.log("Got frame: " + Math.round(count));

   // Capture code here.

   video.requestVideoFrameCallback(frameCounter);
}

video.requestVideoFrameCallback(frameCounter)

This will only fire on new frames, but you may occasionally miss one (which you can detect from a discontinuity in the metadata.presentedFrames count). You might also be slightly late in capturing the frame (usually 16ms, or one call to window.requestAnimationFrame() later than when the video frame is available).

If you're interested in a high level overview of the API, here's a blogpost, or you can take a look at the API's offical github.