Using generators + promises to do "simulated synchronous" communication in/with a Firefox SDK add-on

Question

TL;DR: Is there any way to rewrite this callback-based JavaScript code to use promises and generators instead?

Background

I have a Firefox extension written using the Firefox Add-on SDK. As usual for the SDK, the code is split into an add-on script and a content script. The two scripts have different kinds of privileges: add-on scripts can do fancy things such as, for example, calling native code through the js-ctypes interface, while content scripts can interact with web pages. However, add-on scripts and content scripts can only interact with each other through an asynchronous message-passing interface.

I want to be able to call extension code from a user script on an ordinary, unprivileged web page. This can be done using a mechanism called exportFunction that lets one, well, export a function from extension code to user code. So far, so good. However, one can only use exportFunction in a content script, not an add-on script. That would be fine, except that the function I need to export needs to use the aforementioned js-ctypes interface, which can only be done in an add-on script.

(Edit: it turns out to not be the case that you can only use exportFunction in a content script. See the comment below.)

To get around this, I wrote a "wrapper" function in the content script; this wrapper is the function I actually export via exportFunction. I then have the wrapper function call the "real" function, over in the add-on script, by passing a message to the add-on script. Here's what the content script looks like; it's exporting the function lengthInBytes:

// content script

function lengthInBytes(arg, callback) {
    self.port.emit("lengthInBytesCalled", arg);

    self.port.on("lengthInBytesReturned", function(result) {
        callback(result);
    });
}

exportFunction(lengthInBytes, unsafeWindow, {defineAs: "lengthInBytes",
                                             allowCallbacks: true});

And here's the add-on script, where the "real" version of lengthInBytes is defined. The code here listens for the content script to send it a lengthInBytesCalled message, then calls the real version of lengthInBytes, and sends back the result in a lengthInBytesReturned message. (In real life, of course, I probably wouldn't need to use js-ctypes to get the length of a string; this is just a stand-in for some more interesting C library call. Use your imagination. :) )

// add-on script

// Get "chrome privileges" to access the Components object.
var {Cu, Cc, Ci} = require("chrome");

Cu.import("resource://gre/modules/ctypes.jsm");
Cu.import("resource://gre/modules/Services.jsm");

var pageMod = require("sdk/page-mod");
var data = require("sdk/self").data;
pageMod.PageMod({
    include: ["*", "file://*"],
    attachTo: ["existing", "top"],
    contentScriptFile: data.url("content.js"),
    contentScriptWhen: "start", // Attach the content script before any page script loads.

    onAttach: function(worker) {
        worker.port.on("lengthInBytesCalled", function(arg) {
            let result = lengthInBytes(arg);
            worker.port.emit("lengthInBytesReturned", result);
        });
    }
});

function lengthInBytes(str) {
    // str is a JS string; convert it to a ctypes string.
    let cString = ctypes.char.array()(str);

    libc.init();
    let length = libc.strlen(cString); // defined elsewhere
    libc.shutdown();

    // `length` is a ctypes.UInt64; turn it into a JSON-serializable
    // string before returning it.
    return length.toString();
}

Finally, the user script (which will only work if the extension is installed) looks like this:

// user script, on an ordinary web page
lengthInBytes("hello", function(result) {
    console.log("Length in bytes: " + result);
});

What I want to do

Now, the call to lengthInBytes in the user script is an asynchronous call; instead of returning a result, it "returns" its result in its callback argument. But, after seeing this video about using promises and generators to make async code easier to understand, I'm wondering how to rewrite this code in that style.

Specifically, what I want is for lengthInBytes to return a Promise that somehow represents the eventual payload of the lengthInBytesReturned message. Then, in the user script, I'd have a generator that evaluated yield lengthInBytes("hello") to get the result.

But, even after watching the above-linked video and reading about promises and generators, I'm still stumped about how to hook this up. A version of lengthInBytes that returns a Promise would look something like:

function lengthInBytesPromise(arg) {
    self.port.emit("lengthInBytesCalled", arg);

    return new Promise(
        // do something with `lengthInBytesReturned` event???  idk.
    );
}

and the user script would involve something like

var result = yield lengthInBytesPromise("hello");
console.log(result);

but that's as much as I've been able to figure out. How would I write this code, and what would the user script that calls it look like? Is what I want to do even possible?

A complete working example of what I have so far is here.

Thanks for your help!

Answer 1

A really elegant solution to this problem is coming in the next next version of JavaScript, ECMAScript 7, in the form of async functions, which are a marriage of Promises and generators that sugars over the warts of both. More on that at the very bottom of this answer.

I'm the author of Regenerator, a transpiler that supports async functions in browsers today, but I realize it might be overkill to suggest you introduce a compilation step into your add-on development process, so I'll focus instead on the questions you're actually asking: how does one design a sensible Promise-returning API, and what is the nicest way to consume such an API?

First of all, here's how I would implement lengthInBytesPromise:

function lengthInBytesPromise(arg) {
  self.port.emit("lengthInBytesCalled", arg);

  return new Promise(function(resolve, reject) {
    self.port.on("lengthInBytesReturned", function(result) {
      resolve(result);
    });
  });
}

The function(resolve, reject) { ... } callback is invoked immediately when the promise is instantiated, and the resolve and reject parameters are callback functions that can be used to provide the eventual value for the promise.

If there was some possibility of failure in this example, you could pass an Error object to the reject callback, but it seems like this operation is infallible, so we can just ignore that case here.

So that's how an API creates promises, but how do consumers consume such an API? In your content script, the simplest thing to do is to call lengthInBytesPromise and interact with the resulting Promise directly:

lengthInBytesPromise("hello").then(function(length) {
  console.log(result);
});

In this style, you put the code that depends on the result of lengthInBytesPromise in a callback function passed to the .then method of the promise, which may not seem like a huge improvement over callback hell, but at least the indentation is more manageable if you're chaining a longer series of asynchronous operations:

lengthInBytesPromise("hello").then(function(length) {
  console.log(result);
  return someOtherPromise(length);
}).then(function(resultOfThatOtherPromise) {
  return yetAnotherPromise(resultOfThatOtherPromise + 1);
}).then(function(finalResult) {
  console.log(finalResult);
});

Generators can help reduce the boilerplate here, but additional runtime support is necessary. Probably the easiest approach is to use Dave Herman's task.js library:

spawn(function*() { // Note the *; this is a generator function!
  var length = yield lengthInBytesPromise("hello");
  var resultOfThatOtherPromise = yield someOtherPromise(length);
  var finalResult = yield yetAnotherPromise(resultOfThatOtherPromise + 1);
  console.log(finalResult);
});

This code is a lot shorter and less callback-y, that's for sure. As you can guess, most of the magic has simply been moved into the spawn function, but its implementation is actually pretty straightforward.

The spawn function takes a generator function and invokes it immediately to get a generator object, then invokes the gen.next() method of the generator object to get the first yielded promise (the result of lengthInBytesPromise("hello")), then waits for that promise to be fulfilled, then invokes gen.next(result) with the result, which provides a value for the first yield expression (the one assigned to length) and causes the generator function to run up to the next yield expression (namely, yield someOtherPromise(length)), producing the next promise, and so on, until there are no more promises left to await, because the generator function finally returned.

To give you a taste of what's coming in ES7, here's how you might use an async function to implement exactly the same thing:

async function process(arg) {
  var length = await lengthInBytesPromise(arg);
  var resultOfThatOtherPromise = await someOtherPromise(length);
  var finalResult = await yetAnotherPromise(resultOfThatOtherPromise + 1);
  return finalResult;
}

// An async function always returns a Promise for its own return value.
process(arg).then(function(finalResult) {
  console.log(finalResult);
});

All that's really happening here is that the async keyword has replaced the spawn function (and the * generator syntax), and await has replaced yield. It's not a huge leap, but it will be really nice to have this syntax built into the language instead of having to rely on an external library like task.js.

If you're excited about using async functions instead of task.js, then by all means check out Regenerator!

Answer 2

I think the Promise is built by wrapping your original callback inside the resolve/reject function:

function lengthInBytesPromise(arg) {
    self.port.emit("lengthInBytesCalled", arg);

    let returnVal = new Promise(function(resolve, reject) {
        self.port.on("lengthInBytesReturned", function(result) {
            if (result) { // maybe some kind of validity check
                resolve(result);
            } else {
                reject("Something went wrong?");
            }
        }
    });

    return returnVal;
}

Basically, it'd create the Promise and return it immediately, while the inside of the Promise kicks off and then handles the async task. I think at the end of the day someone has to take the callback-style code and wrap it up.

Your user would then do something like

lengthInBytesPromise(arg).then(function(result) {
    // do something with the result
});