How to edit .webm Blobs captured by Chrome MediaRecorder

Question

In Chrome. I'm using MediaRecorder and canvas.captureStream() to create a webm file of the canvas.

let recorder = new MediaRecorder(document.querySelector('canvas').captureStream(), {mimeType: 'video/webm'});
let chunks = [];
let blob;

recorder.ondataavailable = function(event) {
  if (event.data.size > 0) {
    chunks.push(event.data);
  }
};
recorder.onstop = function() {
  blob = new Blob(chunks, {type: 'video/webm'});
  let url = URL.createObjectURL(blob);
  let a = document.createElement('a');
  document.body.appendChild(a);
  a.href = url;
  a.download = Date.now()+'.webm';
  a.click();
  window.URL.revokeObjectURL(url);
  a.parentNode.removeChild(a);
}
recorder.onstart = function() {
  chunks = [];
}

This is the basic recording and download code along with calling recorder.start() to begin recording and recorder.stop() to end.

The output webm file is fine, the issue I'm having is that because of a shitty computer/overhead, I can't always draw the canvas fast enough to make it a full 60 fps. On the canvas itself, I don't mind a lower framerate, but the lag on drawing to the canvas gets translated into the webm and I'm left with a x0.9 speed video.

I've tried remedying this by using canvas.captureStream(0) to capture only a single frame at a time and matching that up with each canvas render. But this fails because I can't specify the duration each frame should last and the file size becomes enormous since every single frame has all the header information.

I can see in my blob array that the first 131 blobs are constant, and blob 132 has a very large amount of data. After that, there are typically ~7 spacer blobs of 1-byte each and then a single blob containing some larger amount of data. I know the first 132 blobs are header information + my first frame. And I imagine the blobs with larger amounts of data are each frame. I'm also assuming then that the 1-byte spacer blobs are something to do with frame duration or pause for a set amount of time.

What I would like to do is to be able to modify those spacer blobs to specify the exact duration of frame. I've tried to manually do this by copying the 7 spacer blobs between 2 frames where I knew the framerate was ideal, and then removing all other spacers and pasting in these ideal spacer blobs between every frame, but the output file did not play.

Am I misunderstanding the blob data? Is there any way to manually specify the duration a frame lasts by modifying the blob data or am I stuck with whatever framerate I can draw to the canvas?

Answer 1

I was able to define a framerate separate from the canvas refresh rate by pausing and resuming the recorder on timeout, and requesting a frame before pausing again:

let recorder = new MediaRecorder(canvas.captureStream(), {mimeType: 'video/webm'});
recorder.start();
recorder.pause();

function draw() {
  context.drawImage(...);
  recorder.resume();
  setTimeout(function() {
    recorder.requestData();
    recorder.pause();

    //update progress bars or any laggy overhead stuff at this point

    requestAnimationFrame(draw);
  }, 1000/fps);
}
requestAnimationFrame(draw);

This way, any lag in the actual canvas drawing or in updating progress bars etc. will not affect the frame collection of the recorder. recorder.requestData() doesn't seem to be necessary, but also doesn't seem to have any downsides. It's included here for clarity.

I haven't checked in detail but there may be a double frame at the beginning depending on whether or not recorder.start() collects an initial frame and your canvas isn't blank.

Answer 2

I have struggled a lot trying to make videos frame by frame using canvas.captureStream() and MediaRecorder, including using your pause/resume solution. It simply doesn't seem this is intended usage. I have, however, found a library exactly for this purpose: CCapture.js. It worked for me. It works by sampling image dumps of the canvas and joining them afterwards. In the process it overrides some internal functions, so may not be safe for everything(?). But it makes the job very easy. Also, it allows multiple output formats, and in my limited experience, the webm output has much better quality than that achieved with MediaRecorder.