Alternative for the ImageCapture API for better browser support

Question

I use an OCR in order to recognize text from images. I take photos/snapshots from the user's camera/video input using the ImageCapture API from a MediaStream. Here is the code responsible for that:

function getBlobFromMediaStream() {
    const videoTrack = mediaStream.getVideoTracks()[0]

    const imageCapture = new ImageCapture(videoTrack);

    return imageCapture.takePhoto().then(blob => {
        if (!blob) throw "Photo could not be taken";

        return blob;
    })
}

Too bad that this API doesn't work on a lot of browsers (Firefox, IE, Safari). Is there an alternative that can be used?

Answer 1

You could make a fallback with a HTMLVideoElement and a HTMLCanvasElement.
First check if your browser supports the MediaStream Image Capture API by checking if the constructor exists in the window object.

if ('ImageCapture' in window) { 
  // Has support for API.
} else {
  // No support, use fallback.
}

Then based on that either use the API or use your fallback.

In the fallback create a video element, canvas element and a canvas context. Set the MediaStreamTrack as the srcObject of the video element. The video is able to decode the data and make an actual picture.

Use the image of the video to draw an image to the canvas element with the CanvasRenderingContext2D.drawImage() method. With this the canvas element will draw an image of the current frame in the video.

Now you can extract the data that has been drawn on the canvas and turn it into a Blob with the HTMLCanvasElement.toBlob() method.

Wrap the function call inside a Promise for two reasons:

1. You'll need to extract the blob from the callback, a Promise wrapper can help you do this.
2. The ImageCapture.takePhoto() method also returns a Promise. So you'll have the same behavior as you would have when the API is supported.

All put together it would look like this.

function getBlobFromMediaStream(stream) {

  if ('ImageCapture' in window) {

    const videoTrack = stream.getVideoTracks()[0];
    const imageCapture = new ImageCapture(videoTrack);
    return imageCapture.takePhoto();
    
  } else {

    const video = document.createElement('video');
    const canvas = document.createElement('canvas');
    const context = canvas.getContext('2d');

    video.srcObject = stream;

    return new Promise((resolve, reject) => {
      video.addEventListener('loadeddata', async () => {
        const { videoWidth, videoHeight } = video;
        canvas.width = videoWidth;
        canvas.height = videoHeight;

        try {
          await video.play();
          context.drawImage(video, 0, 0, videoWidth, videoHeight);
          canvas.toBlob(resolve, 'image/png');
        } catch (error) {
          reject(error);
        }
      });
    });

  }
  
}

Call your function like in the example below. I've discarded the throw statement so that all rejected states will be caught at the same point when you call the getBlobFromMediaStream function.

According to the docs of ImageCapture.takePhoto(), you either get a Blob when the promise has fulfilled and an error when it has rejected.

Same goes for the fallback. The only difference there is that the HTMLCanvasElement.toBlob() function throws a SecurityError if the bitmap of the canvas is not origin clean.

getBlobFromMediaStream(mediaStream).then(blob => {
  // Work with your blob.
}).catch(error => {
  console.log(`Photo could not be taken. ${error}`);
})