Node JS message queue on Heroku

Question

I need to move my Node JS server running on Heroku to a message queue architecture. Currently, the server receives a HTTP request, does some processing, and responds. The problem is that the processing takes some time, especially when there are lots of requests. This lengthy processing time causes the server to timeout, overload, and crash! My reading tells me a need a background worker to do the processing.

I have zero experience with message queues and background workers and I'm looking for a very simple example to get started. Can anyone suggest a simple, understandable module or example to get started?

I found some examples but they are complex and I'm getting lost! I want a barebones example I can build from.

Answer 1

Let's see how to do this with RabbitMQ. First, you will need a RabbitMQ server to work with in your development environment. If you don't already have it (check "sudo service rabbitmq-server status") you can install (on ubuntu or similar) as follows:

sudo su -c "echo 'deb http://www.rabbitmq.com/debian/ testing main' >> /etc/apt/sources.list"
wget http://www.rabbitmq.com/rabbitmq-signing-key-public.asc
sudo apt-key add rabbitmq-signing-key-public.asc
sudo apt-get update
sudo apt-get install rabbitmq-server
rm  rabbitmq-signing-key-public.asc

Then, get the server running with:

sudo service rabbitmq-server start

You also need to configure a RabbitMQ service for your Heroku deployment. Let's use CloudAMPQ for this example. You can add its Free Plan to your Heroku app with:

heroku addons:create cloudamqp:lemur 

That will create a new CLOUDAMQP_URL environment variable in your Heroku app.

Next, you're going to need a suitable RabbitMQ client for your node.js app. There are a few of them out there, but for this example, let's use ampqlib:

npm install ampqlib --save

That should add something like the following line in your package.json dependencies:

"amqplib": "^0.4.1",

Next thing is to add a background "worker" dyno to your Heroku app. I assume that currently you only have a single Web dyno in your Procfile. So, you need to add another line for instantiating a worker, such as:

worker: node myworker.js

Finally, you need to write the code that will enable your Web dyno to interact with your worker dyno via RabbitMQ.

For the sake of this example, I will assume that your Web dyno will be "publishing" messages to a RabbitMQ message queue, and your worker dyno will be "consuming" these messages.

So, let's start with writing code for publishing to a message queue. This code needs to run somewhere in your Web dyno:

// Define ampq_url to point to CLOUDAMPQ_URL on Heroku, or local RabbitMQ server in dev environment
var ampq_url = process.env.CLOUDAMQP_URL || "amqp://localhost";
var ampq_open = require('amqplib');
var publisherChnl;

function createPublisherChannel() {

    // Create an AMPQ "connection"
    ampq_open.connect(ampq_url)
        .then(function(conn) {
            // You need to create at least one AMPQ "channel" on your connection   
            var ok = conn.createChannel();
            ok = ok.then(function(ch){
                publisherChnl = ch;
                // Now create a queue for the actual messages to be sent to the worker dyno 
                publisherChnl.assertQueue('my-worker-q');
            })
        })
    }

function publishMsg() {
     // Send the worker a message
     publisherChnl.sendToQueue('my-worker-q', new Buffer('Hello world from Web dyno'));
}

You will need to call createPublisherChannel() during the initialisation of your Web dyno. Then, call publishMsg() whenever you want to send a message to the queue.

Finally, let's write the code for consuming the above message in the worker dyno. So, for example, add something like the following in myworker.js:

// Just like in Web dyno...
var amqp_url = process.env.CLOUDAMQP_URL || "amqp://localhost";
var open_ampq = require('amqplib').connect(amqp_url);
var consumerChnl;    

// Creates an AMPQ channel for consuming messages on 'my-worker-q'
function createConsumerChannel() {     
    open_ampq
        .then(function(conn) {
            conn.createChannel()
                .then(function(ch) {
                    ch.assertQueue('my-worker-q');
                    consumerChnl = ch;
            });
        });
}  

function startConsuming() {
    consumerChnl.consume('my-worker-q', function(msg){
        if (msg !== null) {
            console.log(msg.content.toString());
            // Tell RabbitMQ server we have consumed the message
            consumerChnl.ack(msg);
        }
    })
} 

createConsumerChnl().then(startConsuming); 

Finally, test with "heroku local". You should see that you now have 2 processes running in your app, "Web" and "worker". Whenever you call publishMsg() in your Web dyno, you should hopefully see the wroker dyno spit out the message contents to your console. To see what's happening in your RabbitMQ queues, you can use:

sudo rabbitmqctl list_queues