CanvasCaptureMediaStream / MediaRecorder Frame Synchronization

Question

When using CanvasCaptureMediaStream and MediaRecorder, is there a way to get an event on each frame?

What I need is not unlike requestAnimationFrame(), but I need it for the CanvasCaptureMediaStream (and/or the MediaRecorder) and not the window. The MediaRecorder could be running at a different frame rate than the window (possibly at a not regularly divisible rate, such as 25 FPS vs 60 FPS), so I want to update the canvas at its frame rate rather than the window's.

Answer 1

This example currently only fully works on FireFox, since chrome simply stops the canvas stream when the tab is blurred... _{(probably related to this bug, but well, my timer seems to be working but not the recording...)}

[Edit]: it actually now works only in chrome, since they have fixed this bug, but not anymore in FF because of this one (caused by e10s).

---

There doesn't seem to be any event on MediaStream letting you know when a frame has been rendered to it, neither on the MediaRecorder.

Even the currentTime property of the MediaStream (currently only available in FF) doesn't seem to be changing accordingly with the fps argument passed in the captureStream() method.

But what you seem to want is a reliable timer, that won't loose its frequency when i.e the current tab is not focused (which happens for rAF).
Fortunately, the WebAudio API does also have an high precision timer, based on hardware clock, rather than on screen refresh rate.

So we can come with an alternative timed loop, able to keep its frequency even when the tab is blurred.

/*
	An alternative timing loop, based on AudioContext's clock

	@arg callback : a callback function 
		with the audioContext's currentTime passed as unique argument
	@arg frequency : float in ms;
	@returns : a stop function
	
*/
function audioTimerLoop(callback, frequency) {

  // AudioContext time parameters are in seconds
  var freq = frequency / 1000;

  var aCtx = new AudioContext();
  // Chrome needs our oscillator node to be attached to the destination
  // So we create a silent Gain Node
  var silence = aCtx.createGain();
  silence.gain.value = 0;
  silence.connect(aCtx.destination);

  onOSCend();

  var stopped = false;
  function onOSCend() {
    osc = aCtx.createOscillator();
    osc.onended = onOSCend;
    osc.connect(silence);
    osc.start(0);
    osc.stop(aCtx.currentTime + freq);
    callback(aCtx.currentTime);
    if (stopped) {
      osc.onended = function() {
        return;
      };
    }
  };
  // return a function to stop our loop
  return function() {
    stopped = true;
  };
}


function start() {

  // start our loop @25fps
  var stopAnim = audioTimerLoop(anim, 1000 / 25);
  // maximum stream rate set as 25 fps
  cStream = canvas.captureStream(25);

  let chunks = [];
  var recorder = new MediaRecorder(cStream);
  recorder.ondataavailable = e => chunks.push(e.data);
  recorder.onstop = e => {
    // we can stop our loop
    stopAnim();
    var url = URL.createObjectURL(new Blob(chunks));
    var v = document.createElement('video');
    v.src = url;
    v.controls = true;
    document.body.appendChild(v);
  }
  recorder.start();
  // stops the recorder in 20s, try to change tab during this time
  setTimeout(function() {
    recorder.stop();
  }, 20000)
}


// make something move on the canvas
var ctx = canvas.getContext('2d');
var x = 0;
function anim() {
  x = (x + 2) % (canvas.width + 100);
  ctx.fillStyle = 'ivory';
  ctx.fillRect(0, 0, canvas.width, canvas.height);
  ctx.fillStyle = 'red';
  ctx.fillRect(x - 50, 20, 50, 50)
};

btn.onclick = start;

<button id="btn">begin</button>
<canvas id="canvas" width="500" height="200"></canvas>

Nota Bene :
In this example, I set the frequency to 25fps, but we can set it to 60fps and it seems to work correctly even on my old notebook, at least with such a simple animation.