Call a function after all currently scheduled callbacks are executed (node)

Question

I am using a streaming multipart/form-data parser to handle file and field uploads. Whenever a new field or file is parsed an event is fired and I perform some processing. After all fields and files are parsed a 'close' event is fired where I then call a termination function for the request. Here is the setup.

parser.on('fields',function(){

    handleField(callback);

});


parser.on('close',function(){
    dismiss();
})

The problem is that handling a field might take some time which makes the 'close' listener dismiss the request before I have a chance to invoke the callback from the field processing listener.

I have tried to use the setImmediate function which according to its description

queues its callbacks in the event queue after I/O and timer callbacks.

in order to have my dismiss() function be invoked after all currently scheduled callbacks are executed but that didn't work and I have even tried process.nextTick() in case I had the order reversed, but no such luck.

So question is, how could I call my dismiss() function only after all currently scheduled callbacks from the processing function are called?

Answer 1

With Promises and Promise.all, you can make your dismiss a future operation that depends on the resolution of all Promises that get resolved by a wrapper function of your callback(s).

The trick is to wrap your callback into another function that resolves or rejects the wrapper Promise after the callback has been called by the asynchronous (or synchronous) operation. You can store all the Promises in an array and then call Promise.all on that array to yield another Promise to .then() on.

In this example, I'll be assuming that callback is a Node I/O-style handler (i.e. first argument is err) and handleField is an asynchronous Node I/O-style operation that eventually calls callback. From the statement "Whenever a new field or file is parsed an event is fired and I perform some processing.", I'm also making the assumption that 'fields' is an event which occurs N times where N is the number of fields and thus N callbacks must be completed before you can correctly dismiss(). Feel free to comment if my assumptions are incorrect.

var promises = [];

parser.on('fields', function() {
  promises.push(
    new Promise(function(resolve, reject) {

      // wrap the original callback into another function.
      // this function either resolves or rejects the Promise.
      handleField(function(err) {
        // if there's an error,
        // pass it to reject instead of throwing it.
        if (err) {
          reject(err);
        } else {
          // calls your callback with given args.
          // resolves Promise with return value of callback.
          resolve(callback.apply(null, arguments));
        }
      });

    })
  );
});


parser.on('close', function() {
  Promise.all(promises).then(function(values) {
    dismiss(); // All resolved.
  }, function(reason) {
    console.error(reason.stack); // First reject.
    process.exit(); // Exit on error.
  });
});

To learn Promises, you should read the Mozilla documentation or some other reputable reference or tutorial, but I've added some key points below that should help you.

---

new Promise(fn)

When you create a new Promise using the constructor, the function you pass in gets immediately invoked. This is necessary for starting asynchronous tasks immediately and then being able to respond to their results as early as possible using the .then() interface. Within the function that you pass in, you can resolve or reject the Promise however you choose.

---

Using Promise.all

The Promise.all(iterable) method returns a promise that resolves when all of the promises in the iterable argument have resolved, or rejects with the reason of the first passed promise that rejects.

The important thing to note is that the following:

Promise.all([
  new Promise((resolve, reject) => setTimeout(() => resolve('work done'), 3000)),
  new Promise((resolve, reject) => reject('coffee depleted'))
]).then(
  (v) => console.log('all work done!'),
  (r) => console.error(`Error: ${r}`)
)

will immediately reject without waiting for the 1st Promise to resolve because the 2nd Promise gives an early error The first Promise in the array will still eventually resolve, but the point is that Promise.all rejects early, which is good.

---

Shim

If you are running an old version of Node without Promises, you can install the Promises library which is an open source implementation available on GitHub.

npm install promise --save

Then just require it:

var Promise = require('promise');

---

es6-promisify

You could promisify the asynchronous function handleFields assuming that it's an asynchronous Node I/O-style operation that calls callback with err as its first argument:

// npm install es6-promisify --save
var promisify = require("es6-promisify")

handleField = promisify(handleField);
promises.push(
  handleField().then(function(successArgs) {
    // success callback here
  }).catch(function(err) {
    console.error(err);
  })
);

Overall, this looks pretty clean. If you're using the Promise library, then just use Promise.denodify(fn).

Answer 2

One possibility is to use a library that turns NodeJS callback API calls into calls that return promises. Promises are then easy to compose. There are a couple of node-to-promises libs out there, such as promisify. (I haven't used it, can't vouch, just an example.)

But without going down that path, the usual answer is to have both callback call a central "done" method that knows how many calls are outstanding:

var calls = 0;

++calls;
parser.on('fields',function(){

    handleField(callback);
    done();
});

++calls;
parser.on('close',function(){
    done();
})

function done() {
    if (--calls == 0) {
        dismiss();
    }
}

You can even encapsulate that into a utility object:

// ES2015 (ES6); ES5 translation below
class Tracker {
    constructor(callback) {
        this.count = 0;
        this.callback = callback;
    }

    start() {
        ++this.count;
    }

    stop() {
        if (--this.count) {
            this.callback();
        }
    }
}

Then

var track = new Tracker(function() {
    dismiss();
});

track.start();
parser.on('fields',function(){

    handleField(callback);
    track.stop();
});

track.start();
parser.on('close',function(){
    track.stop();
})

And yes, it's a bit fiddly, which is why promises were invented. :-)

---

ES5 translation of Tracker:

function Tracker(callback) {
    this.count = 0;
    this.callback = callback;
}

Tracker.prototype.start = function() {
    ++this.count;
};

Tracker.prototype.stop = function() {
    if (--this.count) {
        this.callback();
    }
};