Draw shapes on HTML5 Canvas...with video

Question

I've been Googling around a bit for an answer and haven't found a definitive one either way: is it possible to play a video using an HTML5 canvas, and also allow the user to draw on this video? The use case, for some context, is to play a video on infinite loop so the user can draw multiple boxes over specific areas to indicate regions of interest.

As a bonus (:P), if I can figure out how to do this on its own, any hints as to how this could be done within Drupal? I'm already looking at the Canvas Field module, but if you have any hints on this point too (though the first one is the priority), that'd be awesome!

Answer 1

You can draw html5 video elements onto a canvas. The drawImage method accepts a video element in the first parameter just like an image element. This will take the current "frame" of the video element and render it onto the canvas. To get fluid playback of the video you will need to draw the video to the canvas repeatedly.

You can then draw on the canvas normally, making sure you redraw everything after each update of the video frame.

Here is a demo of video on canvas

here is a in-depth look into video and the canvas

Answer 2

I recently received this request from a client to provide this feature, and it must be CMS-friendly. The technique involves three big ideas

• a drawing function
• repeatedly calling upon the same drawing function
• using requestAnimationFrame to paint the next frame

Assuming you have a video element already, you'd take the following steps

1. Hide the video element
2. Create a canvas element whose height/width match the video element, store this somewhere
3. Get the context of the canvas element with `canvas.getContext('2d') and also store that somewhere
4. Create a drawing function
5. In that drawing function, you would use canvas.drawImage(src, x, y) where src is the edited version of the current frame of the video;
6. In that drawing function, use recursion to call itself again

I can give you two examples of this being done (and usable for content management systems)

The first is here: https://jsfiddle.net/yywL381w/19/

A company called SDL makes a tool called Media Manager that hosts videos. What you see is a jQuery plugin that takes its parameters from a data-* , makes a request from the Media Manager Rest API, creates a video, and adds effects based entirely on data* attributes. That plugin could easily be tweaked to work with videos called from other sources. You can look at the repo for it for more details on usage.

Another example is here: http://codepen.io/paceaux/pen/egLOeR

That is not a jQuery plugin; it's an ES6 class instead. You can create an image/video and apply a cropping effect with this:

let imageModule = new ImageCanvasModule(module);
imageModule.createCanvas();
imageModule.drawOnCanvas();
imageModule.hideOriginal();

You'll observe, in the ImageCanvasModule class, this method:

drawFrame () {
    if (this.isVideo && this.media.paused) return false;

    let x = 0;
    let width = this.media.offsetWidth;
    let y = 0;

    this.imageFrames[this.module.dataset.imageFrame](this.backContext);
    this.backContext.drawImage(this.media, x, y, width, this.canvas.height);

    this.context.drawImage(this.backCanvas, 0, 0);

    if (this.isVideo) {
        window.requestAnimationFrame(()=>{
            this.drawFrame();
        });
    }
}

The class has created a second canvas, to use for drawing. That canvas isn't visible, it's just their to save the browser some heartache.

The "manipulation" that is content manageable is this.imageFrames[this.module.dataset.imageFrame](this.backContext);

The "frame" is an attribute stored on the image/video (Which could be output by a template in the CMS). This gets the name of the imageFrame, and runs it as a matching function. It also sends in the context (so I can toggle between drawing on the back canvas or main canvas if needed)

then this.backContext.drawImage(this.media, x, y, width, this.canvas.height); draws the image on the back context.

Finally, this appears on the main canvas with this.context.drawImage(this.backCanvas, 0, 0); where I take the backcanvas, and draw it on to the main canvas. So the canvas that's visible has the least amount of manipulations possible.

And at the end, because this is a video, we want to draw a new frame. So we have the function call itself:

        if (this.isVideo) {
        window.requestAnimationFrame(()=>{
            this.drawFrame();
        });

This whole setup allows us to use the CMS to output data-* attributes containing the type of frame the user wants to be drawn around the image. the JavaScript then produces a canvasified version of that image or video. Sample markup might look like:

<video muted loop autoplay data-image-frame="wedgeTop">