How can I write a frame of an embedded YouTube video (via iframe) to a canvas?

Question

I'd like to be able to extract frames from YouTube videos at various points within them (not just at the thumbnails), and do some processing on them. I can embed the video in my website using the iframe API, but I am struggling to find a way to capture that to a canvas. (It's ok if I am forced to capture the entire screen, and ok if I have to make changes to browser settings to allow it.)

Answer 1

One option would be to write a browser extension. But I would guess you would want to avoid that.

The other option is to use the Screen Capture API (which is supported in Chrome, Edge and Firefox). Check out the browser compatibility information on MDN.

Using this API, you can allow your user to share his screen or browser tab and play this on a transient video element:

const videoElem = document.createElement('video');
videoElem.autoplay = true;

const displayMediaOptions = {
  video: {
    cursor: "never"
  },
  audio: false
};

async function startCapture() {
  videoElem.srcObject = await navigator.mediaDevices
    .getDisplayMedia(displayMediaOptions);
}

Later you can draw the video on a canvas to get a Base64 frame (basically a screenshot).

function getScreenshotInBase64() {
  let canvas = document.createElement('canvas');
  let context = canvas.getContext('2d');
  let [w, h] = [videoElem.videoWidth, videoElem.videoHeight];
  canvas.width =  w;
  canvas.height = h;
  context.drawImage(videoElem, 0, 0, w, h);
  return canvas.toDataURL();
}

Limitations with this approach:

• The user may select a different tab / application / screen or the page may be scrolled down and the video not visible.
• You get more than just the video (but based on your description, this is not necessarily bad).
• Pretty scarce browser support (if you consider older or mobile browsers).
• The user has to allow you to do it (basically each time you want to start sharing = recording the video in your case).

I build a small JsFiddle with the above snippets, which gets a Base64 print screen 2 seconds after you start screen sharing (by clicking the start button): https://jsfiddle.net/75Lmbf2o/.