2d Context vs WebGL for Rendering Video

Question

I am currently using the CanvasRenderingContext2D.drawImage() to draw video coming from a RTC mediastream to a canvas. Unfortunately, this takes up considerable CPU resources.

Would it be more performant to do this using a WebGLRenderingContext? (Hardware acceleration?) If yes, how exactly does one handle this, preferably without creating an in-between video element?

Answer 1

No need for 2d Context use WebGL for rendering video if you are already in 3d space.

"Unfortunately, this takes up considerable CPU resources" - I guest you are using special canvas to receive rtc media stream . If canvas is visible somewhere on body you double (cpu) job.

Example for video texture (this code is taken from visual-js project ) You will need a little adapt... See :

export function anyCanvas(path_, nameOfCanvas) {
  var ROOT = this;
  ROOT.iframe = document.createElement('object');
  ROOT.iframe.id = 'canvas2dTextureSurface' + document.getElementsByTagName('object').length;
  // ROOT.iframe.setAttribute('style', 'width:512px;height:512px');

  ROOT.iframe.setAttribute('width', '512');
  ROOT.iframe.setAttribute('height', '512');

  var DIV_CONTENT_STREAMS = document.getElementById('HOLDER_STREAMS');
  DIV_CONTENT_STREAMS.appendChild(ROOT.iframe);

  document.getElementById(ROOT.iframe.id).onload = (event) => {
    ROOT.videoImage = ROOT.iframe.contentDocument.getElementById(nameOfCanvas);
    if(typeof ROOT.iframe.contentWindow.runTextureEditor !== 'undefined') {
      App.scene.outsideBox.streamTextures.iframe.contentWindow.runTextureEditor(nameOfCanvas);
    }
    ROOT.canvasTexture = ROOT.videoImage.getContext('2d');
    // App.scene.outsideBox.streamTextures.iframe.contentWindow.runTextureEditor(nameOfCanvas);
    E('HOLDER_STREAMS').style.display = 'block';
    ROOT.texture = App.tools.loadVideoTexture('glVideoTexture', ROOT.videoImage);
  };

  ROOT.showTextureEditor = function() {
    var T = E('HOLDER_STREAMS').style;
    T.display = 'block';
    T.left = '0';
  };

  ROOT.iframe.data = path_;

}   
 

Try and see : https://github.com/zlatnaspirala/matrix-engine

Examples: https://maximumroulette.com/apps/matrix-engine/examples-build.html

Video example:

https://maximumroulette.com/apps/matrix-engine/query-build.html?u=camera_texture