WebGL: Is there an efficient way to upload only part of an image/canvas as a texture?

Question

I am working on a 2D layer-based application, and I'd like to do the compositing using WebGL. The layers might be shifted relative to each other, and each frame only a small (rectangular) part of each layer may change. However, the width and height of that rectangular part may vary unpredictably. I would like to use one canvas (2D) and one texture per layer, and each frame redraw on each canvas only the part of the layer that has been modified, then simply upload that small area to the GPU to update the corresponding part to texture, before the GPU makes the compositing for me. However I have not found an efficient way to upload just a part of an image to a part of a texture. It seems that texSubImage2D() can update a part of a texture, but only takes full images/canvases, and it does not seem to be possible to specify a rectangular area of the image to use.

I have thought of a few ways of doing this, but each seems to have obvious overhead:

• use getImageData() + texSubImage2D() to only upload to the GPU the part that changes (overhead in converting the canvas pixel data to ImageData)
• re-upload the whole layer canvas each frame with texImage2D()
• or create/resize a small canvas2D to the correct dimension to fit perfectly for each layer modification, then use texSubImage2D() to send it to update the related texture (memory reservation overhead)

So, is there a way to specify a part of an image/a canvas for the texture ? Something like texImagePart2D() and texSubImagePart2D, which would both accept four more parameters, sourceX, sourceY, sourceWidth and sourceHeight to specify the rect area of the image/canvas to use ?

Answer 1

Unfortunately no there is no way to upload a portion of a canvas/image.

OpenGL ES 2.0 on which WebGL is based does not provide a way to do that. OpenGL ES 3.0 does provide a way to upload smaller rectangle of the source to a texture or portion of a texture so maybe the next version of WebGL will provide that feature.

For now you could have a separate canvas to help upload. First size the canvas to the match the portion you want to upload

canvasForCopying.width = widthToCopy;
canvasForCopying.height= heightToCopy;

Then copy the portion of the canvas you wanted to copy to the canvas for copying

canvasForCopying2DContext.drawImage(
    srcCanvas, srcX, srcY, widthToCopy, heightToCopy,
    0, 0, widthToCopy, heightToCopy);

Then use that to upload to the texture where you want it.

gl.texSubImage2D(gl.TEXTURE_2D, 0, destX, destY, gl.RGBA, gl.UNSIGNED_BYTE, 
                 canvasForCopying);

getImageData will likely be slower because the browser has to call readPixels to get the image data and that stalls the graphics pipeline. The browser does not have to do that for drawImage.

As for why texImage2D can sometimes be faster than texSubImage2D it depends on the driver/GPU but apparently sometimes texImage2D can be implemented using DMA whereas texSubImage2D can not. texImage2D can also be pipelined by making a new texture and lazily discarding the old one where as texSubImage2D can't.

Personally I wouldn't worry about it but if you want to check, time uploading tens of thousands of textures using both texImage2D and texSubImage2D (don't time just one as graphics are pipelined). You'll probably find texSubImage2D is faster if your texture is large and the portion you want to update is smaller than 25% of the texture. At least that's what I found last time I checked. Most current drivers are at least optimized in that if you call texSubImage2D and happen to be replacing the entire contents they'll call the texImage2D code internally.

update

There's a couple of things you can do

1. For images you can use fetch and ImageBitmap to load a portion of an image into an ImageBitamp which you can then upload that. Example of getting a portion of an image

   In the example below we call fetch, then get a Blob and use that blob to make an ImageBitmap with only a portion of the image. The result can be passed to texImage2D but in the interest of brevity the sample just uses it in a 2d canvas.

fetch('https://i.imgur.com/TSiyiJv.jpg', {mode: 'cors'})
.then((response) => {
  if (!response.ok) {
    throw response;
  }
  return response.blob();
})
.then((blob) => {
  const x = 451;
  const y = 453;
  const width = 147;
  const height = 156;
  return createImageBitmap(blob, x, y, width, height);
}).then((bitmap) => {
  useit(bitmap);
}).catch(function(e) {
  console.error(e);
});

// -- just to show we got a portion of the image

function useit(bitmap) {
  const ctx = document.createElement("canvas").getContext("2d");
  document.body.appendChild(ctx.canvas);
  ctx.drawImage(bitmap, 0, 0);
}

2. In WebGL2 there are the gl.pixelStorei settings

   UNPACK_ROW_LENGTH // how many pixels a row of the source is UNPACK_SKIP_ROWS // how many rows to skip from the start of the source UNPACK_SKIP_PIXELS // how many pixels to skip from the left of the source

   so using those 3 settings you can tell webgl2 the source is wider but the portion you want from it is smaller. You pass a smaller width to texImage2D and the 3 settings above help tell WebGL how to get out the smaller portion and where to start.