getImageData - Web workers - How can I reduce garbage collection?

Question

I've got a sample web-worker canvas update script running, but I've noticed it stalls every few seconds for about 200ms. The usual loop time is around 15ms.

I'm guessing this is garbage collection - it looks like it from the profiler.

http://codepen.io/SarahC/pen/bgBoMM

I think it is in this function:

function nextFrame(){
  timeChart.start();
  workersWorking = workerCount;
  var stripHeight = ~~( h / workerCount );
  for(var i = 0; i < workerCount; i++){
    var localImageData = ctx.getImageData(0, stripHeight * i, w, stripHeight); /// This needs putting in constant memory.... GC takes ages here.
    workers[i].postMessage({imageData: localImageData, YPosition: stripHeight * i, threadNumber: i});
  }
}

If it is this bit that's making all the garbage memory, I don't know what I can do to keep using the same area of memory for these blocks of data.

Answer 1

Pass the buffer of your imageData instead of the imageData itself.

This way, your buffer is transferred (with a zero-copy operation), and doesn't pollute main thread's memory anymore.
Otherwise, when you don't transfer it your object is structured cloned (just like if you did JSON.parse(JSON.stringify(yourObject));), which means that you computer holds three copies of the same data in memory when you send back from the worker to the main thread.

Note that when being passed in the worker, the imageData's data is not available anymore in the main thread (it will throw an error if i.e you try to putImageData() it). I unfortunately don't know of a good way to change the buffer of an ImageData, but you can set a buffer at the creation thanks to the ImageData() constructor (apparently still not supported in IE...), which will indeed just create a pointer to the arrayBuffer.

So when all this is supported, only the ImageData structure (basically an object {width:XXX, height:XXX}) is created instead of an heavy buffer. Everything else is just moved, and doesn't pollute the memory.

let workerURL = URL.createObjectURL(new Blob([workerScript.textContent], {
  type: 'application/javascript'
}));

const worker = new Worker(workerURL);
worker.onmessage = e => {
  let buf = e.data,
    arr = new Uint8ClampedArray(buf),
    processedImageData;
  try {
    processedImageData = new ImageData(arr, imageData.width, imageData.height);
  } catch (e) {
    processedImageData = ctx.createImageData(imageData.width, imageData.height);
    processedImageData.data.set(arr);
  }
  // checks that we didn't created an useless buffer in this last step
  // IE will because it doesn't support new ImageData(buf)
  console.log('Does our TypedArray share the same buffer as the one we received ? ',
              arr.buffer === buf);
  console.log('Does our new imageData share the same buffer as the one we received ? ',
              processedImageData.data.buffer === buf);
  // Note that here a check for the original imageData's buffer has no sense
  //       since it has been emptied
  ctx.putImageData(processedImageData, 0, 0);
}

const ctx = canvas.getContext('2d');
ctx.fillStyle = 'green';
ctx.fillRect(20, 20, 60, 80);
let imageData = ctx.getImageData(0, 0, 300, 150);
// pass it as transferable
worker.postMessage(imageData.data.buffer, [imageData.data.buffer]);
console.log(imageData.data.length, 'now empty')

<script type="worker-script" id="workerScript">
	self.onmessage = e => {
		let buf = e.data,
		arr = new Uint8Array(buf);
		console.log('worker received', buf);
		for(let i =0; i<arr.length; i+=4){
			arr[i] = (arr[i] + 128) % 255;
			arr[i+1] = (arr[i+1] + 128) % 255;
			arr[i+2] = (arr[i+2] + 128) % 255;			
			}
		self.postMessage(buf, [buf]);
		// this won't print in stacksnippet's console
		// you have to check your dev tools' one
		console.log('worker now holds', buf.byteLength, 'empty');

		};
</script>
<canvas id="canvas"></canvas>

And a counter example using structure cloning :

let workerURL = URL.createObjectURL(new Blob([workerScript.textContent], {
  type: 'application/javascript'
}));

const worker = new Worker(workerURL);
worker.onmessage = e => {
  let buf = e.data;
  // so our original imageData's arrayBuffer is still available
  imageData.data.set(buf);
  // Here we can check for equality with the first arrayBuffer
  console.log('Is the first bufferArray the same as the one we received ?', imageData.data.buffer === buf);  
  ctx.putImageData(imageData, 0, 0);
}

const ctx = canvas.getContext('2d');
ctx.fillStyle = 'green';
ctx.fillRect(20, 20, 60, 80);
let imageData = ctx.getImageData(0, 0, 300, 150);
// pass it as transferable
worker.postMessage(imageData.data.buffer);
console.log(imageData.data.length, 'not empty')

<script type="worker-script" id="workerScript">
	self.onmessage = e => {
		let buf = e.data,
		arr = new Uint8Array(buf);
		console.log('worker received', buf);
		for(let i =0; i<arr.length; i+=4){
			arr[i] = (arr[i] + 128) % 255;
			arr[i+1] = (arr[i+1] + 128) % 255;
			arr[i+2] = (arr[i+2] + 128) % 255;			
			}
		console.log(arr);
		self.postMessage(buf);
		// this won't print in stacksnippet's console
		// you have to check your dev tools' one
		console.log('worker now holds', buf.byteLength, 'full');
		};
</script>
<canvas id="canvas"></canvas>

Answer 2

You can only reduce the overall GC hits in each JS context by reusing the pixel data arrays, but you can not make a significant change. The problem is the data that is transferred to and from the workers.

When you post data to the worker, the worker has to assign memory to receive that data. It can not say "here is some RAM I am already using please put it in here". Sadly no, a posted message and its contents arrive at the worker's context as a new object every time. And the same applies to the returning data, every message is a new message, with a new referance, assignment and eventual removal.

You can send smallers chunks of data at a time, and that may spread the GC hits out so you don't get the big GC spikes. But memory usage is tied to memory throughput, unless you reduce the throughput you will not reduce the GC load.

You might want to have a look at sharedArrayBuffers as they provide a shared memory resource between Javascript threads. Current support is Chrome (requires flags) and Firefox. But there is a push to get this type of memory managment happening so it may be worth your while trying it out.

SharedArrayBuffers should go a long way to almost eliminating GC hits for this type of application.

UPDATE:

In the light of new information you can try using the transferable transfer argument in the worker.postMessage call. See the draft W3C worker postmessage for posting to the worker and worker global scope postmessage for returning data from the worker.

The transferable object is defined here transferable objects which points out that you can only transfer an object once. When the worker receives the object it can not return it as a transferable object (according to the documentation). You will have to create a new typed array from any array you receive if you wish to return it as a transferable.

What is unclear from the documentation is how this effect memory management within the context of the sender.

Update 2

After playing with transferable data I have found that to stop any memory overhead you need to have two copies of the data (in the case of imageData)

The imageData.data property that holds the pixel data can not be used after it has been transfered. You need to either create a new imageData array or copy the data to send to another array and copy it back when returned with the typedArray set function.

The following is an example (bits of code not the whole thing) of using transferable data to transfer data to and from a worker while not incurring undue GC overheads. Running code similar on Chrome and checking the timeline shows no GC hits over 0.002ms

// one time set up
var imgData = ctx.getImageData(0, 0, 512, 512);
var tData = new Uint8ClampedArray(512 * 512 * 4); 
tData.set(imgData.data);

// repeats from here
worker.sendMessage(tData.buffer, [tData.buffer]);

// In the worker
onmessage(event){
    var data = new Uint8ClampedArray(event.data);
    // process the data
    // return data
    postMessage(event.data,[event.data]);
}

// Back on the main thread
onmessage(event){
    tData = new Uint8ClampedArray(event.data);
    imgData.data.set(tData);
    ctx.putImageData(imgdata, 0, 0);
}
// now you can resend tData as it is a new typedArray referance (to the same data)