Web worker and scaling images

Question

I need to scale images in array form in a Web Worker. If I was outside a web worker I could use a canvas and drawImage to copy certain parts of an image or scale it.

Look like in a web worker I can't use a canvas so, what can I do? Is there any pure Javascript library that can help me?

Thanks a lot in advance.

Answer 1

Scaling can be done in various ways, but they all boil down to either removing or creating pixels from the image. Since images are essentially matrices (resized as arrays) of pixel values, you can look at scaling up images as enlarging that array and filling in the blanks and scaling down images as shrinking the array by leaving values out.

That being said, it is typically not that difficult to write your own scale function in JavaScript that works on arrays. Since I understand that you already have the images in the form of a JavaScript array, you can pass that array in a message to the Web Worker, scale it your scale function and send the scaled array back to the main thread.

In terms of representation I would advise you to use the Uint8ClampedArray which was designed for RGBA (color, with alpha channel) encoded images and is more efficient than normal JavaScript arrays. You can also easily send Uint8ClampedArray objects in messages to your Web Worker, so that won't be a problem. Another benefit is that a Uint8ClampedArray is used in the ImageData datatype (after replacing CanvasPixelArray) of the Canvas API. This means that it quite easy to draw your scaled image back on a Canvas (if that was what you wanted), simply by getting the current ImageData of the canvas' 2D context using ctx.getImageData() and changing its data attribute to your scaled Uint8ClampedArray object.

By the way, if you don't have your images as arrays yet you can use the same method. First draw the image on the canvas and then use the data attribute of the current ImageData object to retrieve the image in a Uint8ClampedArray.

Regarding scaling methods to upscale an image, there are basically two components that you need to implement. The first one is to divide the known pixels (i.e. the pixels from the image you are scaling) over the larger new array that you have created. An obvious way is to evenly divide all the pixels over the space. For example, if you are making the width of an image twice as wide, you want simply skip a position after each pixel leaving blanks in between.

The second component is then to fill in those blanks, which can be slightly less straightforward. However, there are several that are fairly easy. (On the other hand, if you have some knowledge of Computer Vision or Image Processing you might want to look at some more advanced methods.) An easy and somewhat obvious method is to interpolate each unknown pixel position using its nearest neighbor (i.e. the closest pixel value that is known) by duplicate the known pixel's color. This does typically result in the effect of bigger pixels (larger blocks of the same color) when you scale the images too much. Instead of duplicating the color of the closest pixel, you can also take the average of several known pixels that are nearby. Possibly even combined with weights were you make closer pixels count more in the average than pixels that are farther away. Other methods include blurring the image using Gaussians. If you want to find out what method is the best for your application, look at some pages about image interpolation. Of course, remember that scaling up always means filling in stuff that isn't really there. Which will always look bad if you do it too much.

As far as scaling down is concerned, one typically just removes pixels by transferring only a selection of pixels from the current array to the smaller array. For example if you would want to make the with of an image twice as small, you roughly iterate through the current array with steps of 2 (This depends a bit on the dimensions of the image, even or odd, and the representation that you are using). There are methods that do this even better by removing those pixels that could be missed the most. But I don't know enough about them.

By the way, all of this is practically unrelated to web workers. You would do it in exactly the same way if you wanted to scale images in JavaScript on the main thread. Or in any other language for that matter. Web Workers are however a very nice way to do these calculations on a separate thread instead of on the UI thread, which means that the website itself does not seem unresponsive. However, like you said, everything that involves the canvas element needs to be done on the main thread, but scaling arrays can be done anywhere.

Also, I'm sure there are JavaScript libraries that can do this for you and depending on their methods you can also load them in your Web Worker using importScripts. But I would say that in this case it might just be easier and a lot more fun to try to write it yourself and make it tailor-made for your purpose.

And depending on how advanced your programming skills are and the speed at which you need to scale you can always try to do this on the GPU instead of on the CPU using WebGL. But that does seem a slight overkill in this case. Also, you can try to chop your image in several pieces and try to scale the separate parts on several Web Workers making it multi-threaded. Although it is certainly not trivial to combine the parts later. Perhaps multi-threaded makes more sense when you have a lot of images that need to be scaled on the client side.

It all really depends on your application, the images and your own skills and desires.

Anyway, I hope that roughly answers your question.

Answer 2

I feel some specifics on mslatour's answer are needed, since I just spent 6 hours trying to figure out how to "…simply… change its data attribute to your scaled Uint8ClampedArray object". To do this:

① Send your array back from the web-worker. Use the form:

self.postMessage(bufferToReturn, [bufferToReturn]);

to pass your buffer to and from the web worker without making a copy of it, if you don't want to. (It's faster this way.) (There is some MDN documentation, but I can't link to it as I'm out of rep. Sorry.) Anyway, you can also put the first bufferToReturn inside lists or maps, like this:

self.postMessage({buffer:bufferToReturn, width:500, height:500}, [bufferToReturn]);

You use something like

webWorker.addEventListener('message', function(event) {your code here})

to listen for a posted message. (In this case, the events being posted are from the web worker and the event doing the listening is in your normal JS code. It works the same other way, just switch the 'self' and 'webWorker' variables around.)

② In your browser-side Javascript (as opposed to worker-side), you can use imageData.data.set() to "simply" change the data attribute and put it back in the canvas.

var imageData = context2d.createImageData(width, height);
imageData.data.set(new Uint8ClampedArray(bufferToReturn));
context2d.putImageData(imageData, x_offset, y_offset);

I would like to thank hacks.mozilla.org for alerting me to the existence of the data.set() method.

p.s. I don't know of any libraries to help with this… yet. Sorry.

Answer 3

I have yet to test it out myself, but there is a pure JS library that might be of use here:

https://github.com/taisel/JS-Image-Resizer