Queue tasks in Node

Question

I'm building a node application that will act as a "worker" to scaffold out a new application and then upload the application to AWS. There are 5 tasks that complete the scaffold cycle.

I'd like to know if it's possible in Node/Express to queue incoming requests and then start the scaffold cycle for requests in the queue when the cycle has successfully completed. There should only be one scaffold cycle running at one time.

Answer 1

Yes, you can do that. Perhaps your current code looks something like this (I'm assuming you're using promises):

app.get('/', function (req, res) {
  scaffold().then(function() {
    res.send('done');
  });
});

We'll use promise-queue to simplify things. First we need to create a queue:

var queue = new Queue(1);

The argument is the number of items the queue will run concurrently. Since you want no concurrency, we use 1. Now, we need to pass in a promise factory instead of running a promise:

app.get('/', function (req, res) {
  queue.add(function() {
    return scaffold().then(function() {
      res.send('done');
    });
  });
});

If we just passed in a promise, it would start immediately. That' why we must pass in a function that returns the promise.

If you want to respond to the request immediately and not wait for the task to finish, you can move that outside of the promise factory. This also allows us to avoid the extra anonymous function since now scaffold is the promise factory that we want to queue.

app.get('/', function (req, res) {
  queue.add(scaffold);
  res.send('done');
});

Answer 2

This isn't the "strictly node.js" answer, but I'd highly recommend looking into Beanstalkd as a way of solving this problem (using fivebeans as the nodejs library).

Put each of your tasks in a named job queue, only pull a new job out when you're ready to work on it, and put the next step in the next queue when it's ready. We use this in production to do a complicated series of transcoding steps every time a new video is uploaded, where each step depends on one or more previous steps having completed.