HTML5 frame-by-frame viewing / frame-seeking?

Question

I'm looking for a way to view HTML5 <video>, frame-by-frame.

The scenario: Having a video, with an additional button that skips to the next frame when pressed.

What do you think is the best method to do this?

1. Play the video normally, Listen to timeUpdate event, which on FireFox is called for every frame, and then just pause the video. However, the other browsers don't behave like Firefox.
2. Change the currentTime element manually to +1/24 of a second, where "24" is the frame rate. I have no idea how to aquire the FPS, however.
3. Any other helpful way you can think of.

---

EDIT

I have found this very useful HTML5 test page, which tracks all browsers' ability to achieve accurate frame-seeking.

Answer 1

It seems that most browsers allow the second approach, although you would need to know the frame rate. Opera, however, is the exception, and requires an approach similar to your first one (the result is not perfect). Here's a demo page I came up with that uses a 29.97 frames/s video (U.S. television standard). Note that it has not been extensively tested, so it might not work in IE 9, Firefox 4, or future versions of any browser.

HTML:

<p id="time"></p>
<video id="v0" controls tabindex="0" autobuffer preload>
    <source type="video/webm; codecs=&quot;vp8, vorbis&quot;" src="http://www.html5rocks.com/tutorials/video/basics/Chrome_ImF.webm"></source>
    <source type="video/ogg; codecs=&quot;theora, vorbis&quot;" src="http://www.html5rocks.com/tutorials/video/basics/Chrome_ImF.ogv"></source>
    <source type="video/mp4; codecs=&quot;avc1.42E01E, mp4a.40.2&quot;" src="http://www.html5rocks.com/tutorials/video/basics/Chrome_ImF.mp4"></source>
    <p>Sorry, your browser does not support the &lt;video&gt; element.</p>
</video>

JavaScript (run on page load and uses jQuery 1.4.4 for the sake of brevity):

var vid = $('#v0')[0];

vid.onplay = vid.onclick = function() {
    vid.onplay = vid.onclick = null;

    setTimeout(function() {
        vid.pause();
        setInterval(function() {
            if($.browser.opera) {
                var oldHandler = vid.onplay;
                vid.onplay = function() {
                    vid.pause();
                    vid.onplay = oldHandler;
                };
                vid.play();
            } else {
                vid.currentTime += (1 / 29.97);
            }
        }, 2000);
    }, 12000);

    setInterval(function() {
        $('#time').html((vid.currentTime * 29.97).toPrecision(5));
    }, 100);
};

Answer 2

Having just been fighting this very same problem I cam up with a brute force method to find the frame rate.Knowing that the frame rate will never pass 60 frames a second I seek through the video at 1/60th second steps. Comparing each frame to the last by putting the video onto a canvas element and then using getImageData to get pixel data. I do this over a second of video and then count up the total number of unique frames to get the frame rate.

You do not have to check every single pixel. I Grab about 2/3rds of the inside of the video and then check about every 8th pixel across and down (depending on the size). You can stop the compare with just a single pixel different so this method in reasonably fast and reliable, well not compared to say reading a frameRate property, but better than guessing.

Answer 3

The best thing you can probably do until standards are confirmed and implemented (that'll be ages!) is take a guess at the frame-rate and increment by that time. Unless you're in a controlled environment, I'd advise against relying heavily on HTML5... as much as I love its features, it won't be reliably supported.

Answer 4

This is my little keyPress handler I use to seek videos that don't have controls.
If the video is paused it skips by frames instead.

const video = document.querySelector('#yourVideo')
const expectedFramerate = 60 // yourVideo's framerate

function handleKey(ev) {
  let d = 0;

  switch (ev.key) {
    case ",": d = -5; break;  // normal
    case ".": d = +5; break;
    case "?": d = -10; break; // shift
    case ":": d = +10; break;
    case "<": d = -2; break;  // rightAlt
    case ">": d = +2; break;
    case " ": togglePlayback(); break;
  }

  if (d) {
    if (video.paused) video.currentTime += Math.sign(d) * 1/expectedFramerate
    else video.currentTime += d
  }
}

document.onkeypress = handleKey

function togglePlayback() {
  video.paused
    ? video.play()
    : video.pause()
}

Note, you could easily add a keybinding to increase/decrease/switch the expectedFramerate on the fly to suit your needs.

---

Bonus: Add the script as Bookmarklet

https://gist.github.com/ackvf/b180e9883069ad753969a30cb2622787

Answer 5

Check this:

https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/seekToNextFrame

It's experimental at this moment and shouldn't be used on production site but looks like up-to-date solution.

It works for Firefox 56 for now

var seekCompletePromise = HTMLMediaElement.seekToNextFrame();

HTMLMediaElement.seekToNextFrame();