RabbitMQ for NodeJS with Express routing

Question

My server is running NodeJS and uses the amqplib api to request data from another application. The NodeJS server is receiving the information successfully but there's a noticable delay and I'm trying to determine whether I am doing this in the most efficient manner. Specifically I'm concerned with the way that I open and close connections.
Project Layout
I have two controller files that handle receiving and requesting the data, request.img.server.controller.js and receive.img.server.controller.js. Finally the routes handle the controller methods when a button on the front end is pushed, oct.server.routes.js.

request.img.server.controller.js

'use strict';

var amqp = require('amqplib/callback_api');
var connReady = false;
var conn, ch;
amqp.connect('amqp://localhost:5672', function(err, connection) {
    conn = connection;
    connReady = true;
    conn.createChannel(function(err, channel) {
        ch = channel;
    });
});


exports.sendRequest = function(message) {
    console.log('sending request');

    if(connReady) {
        var ex = '';
        var key = 'utils';

        ch.publish(ex, key, new Buffer(message));
        console.log(" [x] Sent %s: '%s'", key, message);
    }
};

receive.img.server.controller.js

var amqp = require('amqplib/callback_api');
var fs = require('fs');
var wstream = fs.createWriteStream('C:\\Users\\yako\\desktop\\binarytest.txt');

var image, rows, cols;
exports.getResponse = function(resCallback) {
    amqp.connect('amqp://localhost:5672', function(err, conn) {
        conn.createChannel(function(err, ch) {
            var ex = '';

            ch.assertQueue('server', {}, function(err, q) {
                console.log('waiting for images');
                var d = new Date();
                var n = d.getTime();
                ch.consume(q.queue, function(msg) {
                    console.log(" [x] %s: '%s'", msg.fields.routingKey, msg.content.toJSON());
                    rows = msg.content.readInt16LE(0);
                    cols = msg.content.readInt16LE(2);
                    console.log("rows = %s", msg.content.readInt16LE(0));
                    console.log("cols = %s", msg.content.readInt16LE(2));
                    image = msg.content;
                    var currMax = 0;
                    for (var i = 4; i < image.length; i+=2) {
                        if (image.readInt16LE(i) > currMax) {
                            currMax = image.readInt16LE(i);
                        }
                        wstream.write(image.readInt16LE(i) + ',');
                    }
                    console.log('done writing max is', currMax);
                    //console.log(image);
                    resCallback(rows, cols, image);
                }, {
                    noAck: true
                });
            });
        });
    });
};

oct.server.routes.js

'use strict';

module.exports = function(app) {
    var request_img = require('../../app/controllers/image-tools/request.img.server.controller.js');
    var receive_img = require('../../app/controllers/image-tools/receive.img.server.controller.js');

    // oct routes
    app.get('/load_slice', function(req, res) {
        console.log('load slice hit');
        receive_img.getResponse(function (rows, cols, image) {
            res.end(image);
        });
        request_img.sendRequest('123:C:\\Users\\yako\\Documents\\Developer\\medicaldiag\\test_files\\RUS-01-035-09M-21.oct');
    });
};

Answer 1

The way you're opening connections is bad, and is at least part of the performance problem.

Connections are expensive to open. They open a new TCP/IP connection on a TCP/IP port between the client and rabbitmq server. This takes time, and uses up a limited resource on both the client and server.

Because of this, a single connection to RabbitMQ should be created and used within each of your node.js processes. This one connection should be shared by all of the code in that process.

Whenever you need to do something with RabbitMQ, open a new channel on the shared connection and do your work. Channels are cheap and are meant to be opened and closed as needed, within a connection.

More specifically in your code, the receive.img.server.controller.js file is the major problem. This opens a new connection to RabbitMQ every time you call the getResponse method.

If you have 10 users hitting the site, you'll have 10 open RabbitMQ connections when 1 would be sufficient. If you have thousands of users hitting the site, you'll have thousands of open RabbitMQ connections when 1 would be sufficient. You also run the risk of exhausting your available TCP/IP connections on the RabbitMQ server or client.

Your receive.img.server.controller.js should look more like your request.img.server.controller.js - one connection open, and re-used all the time.

---

Also, FWIW - I recommend using the wascally library for RabbitMQ w/ node.js. This library sits on top of amqplib, but makes things significantly easier. It will manage your one connection for you, and make it easier for you to send and receive messages.

I also have some training material available for RabbitMQ and node.js that covers the basics of amqplib and then moves in to using wascally for real application development.