How to transfer large objects using postMessage of webworker?

Question

I have read that transferable objects can be transferred really fast using postmessage of web worker. According to this transferable objects are either arraybuffer or messageport.

Question is, how do I convert say an arbitrary object that is of large size (30 mb) to a transferable object and pass it as an argument to postmessage. From what I understand I can convert my array to json string and then convert json string to raw byte data and store that inside of an array object. However, this seems to defeat the purpose of fast transferring.

could someone enlighten me to pass an object as transferable object or if it's even possible?

Thanks in advance!

Answer 1

This misconception is quite recurring here. You're imagining that it's possible to write some fast javascript code to convert your large object into transferable. But indeed, any conversion code you write defeats the purpose, just as you said. And the more complex data, the more speed you lose.

Objects are normally (when not transfering) converted by native structured clone algorithm (which uses implementation defined format and sure is optimal). Any javascript code you write will most likely be slower than structured clone, while achieving the same goal - transferring data as binary.

The purpose of transferable objects is to allow transfer for binary data, such as images (from canvas), audio or video. These kinds of data can be transferred without being processed by structured clone algorithm, which is why transferable interface was added. And the effect is insignificant even for these - see an answer about transferable speed.

As a last note, I wrote a prototype based library that converts javascript objects to ArrayBuffer and back. It's slower, especially for JSON like data. It's advantages (and advantages of any similar code you write) are:

1. You can define custom object conversions
2. You can use inheritance (eg. sending your own types, like Foo)

Code to transfer JSON like object

If your data is like JSON, just stick to structured clone and do not transfer. If you don't trust me, test it with this code. You will see it's slower than normal postMessage.

var object = {dd:"ddd", sub:{xx:"dd"}, num:666};
var string = JSON.stringify(object);
var uint8_array = new TextEncoder(document.characterSet.toLowerCase()).encode(string);
var array_buffer = uint8_array.buffer;
// now transfer array buffer
worker.postMessage(array_buffer, [array_buffer])

The opposite conversion, considering you have some ArrayBuffer:

// Let me just generate some array buffer for the simulation
var array_buffer = new Uint8Array([123,34,100,100,34,58,34,100,100,100,34,44,34,115,117,98,34,58,123,34,120,120,34,58,34,100,100,34,125,44,34,110,117,109,34,58,54,54,54,125]).buffer;
// Now to the decoding
var decoder = new TextDecoder("utf-8");
var view = new DataView(array_buffer, 0, array_buffer.byteLength);
var string = decoder.decode(view);
var object = JSON.parse(string);

Answer 2

Should have looked up Tomas's answer earlier.

Proof, although not specifically the way Tomas suggested.

Version A

Version B

I manually converted to a stringified json obejct to a Uint8Array like so:

function stringToUintArray(message) {
  var encoded = self.btoa(message);
  var uintArray = Array.prototype.slice.call(encoded).map(ch => ch.charCodeAt(0));
  var uarray = new Uint8Array(uintArray);
  return uarray;
}

and transferred it like so from the web worker to main thread:

console.time('generate');
    var result = generate(params.low, params.high, params.interval, params.size);
    var uarr = stringToUintArray(JSON.stringify(result));
    console.timeEnd('generate');
    self.postMessage(uarr.buffer, [uarr.buffer]);

and on the main thread I did something like this:

var uarr = new Uint8Array(e.data);
var json = UintArrayToString(uarr);
var result = JSON.parse(json);