node.js + socket.io + redis architecture - horizontal serverscaling socket connections?

Question

i'm using node.js for the first time and hoping for an advice:

i installed the following programs on my server:

• node.js v0.11.3-pre
• express v3.3.4
• socket.io v0.9.14
• connect-redis v1.4.5
• Redis server v=2.6.14
• redis-cli 2.6.14

First of all, i created an express app:

express testApplication

In the created "package.json" i defined all neccessary depencies.

From the start i defined a cluster for vertically scaling (multi-processes) in a file called "cluster.js":

var cluster = require('cluster');

if( cluster.isMaster ) {
        var noOfWorkers = process.env.NODE_WORKERS || require('os').cpus().length;

        console.log("Workers found: " + noOfWorkers);

        for (var i = 0; i < noOfWorkers; i += 1) {
                cluster.fork();
        }
} else {
        require('./app.js');
}

cluster.on('exit', function(worker, code, signal) {
        var exitCode = worker.process.exitCode;

        console.log('worker' + worker.process.pid + ' died (' + exitCode + '). restarting...');

        if( typeof cluster.workers[worker.id] != "undefined" )
                cluster.workers[worker.id].delete();

        cluster.fork();
});

In my "app.js" file i defined REDIS for socket.io storing:

  io.set('store', new RedisStore({
            redisPub: pub,
            redisSub: sub,
            redisClient: client
          }));

So far, so good, all this works pretty nice.

When a client connects to the socket.io server, the cluster handles the connections with different workers.

My intention is, that a client can send a message to a specific another client, so the socket.io server have to find the socket from the receipient to send the message only to this user. The solution for me is, that i store all created socket ids for every user in an array and when sending a message, i select the relevant socket ids in the array, gets the sockets by id, and send the message to the socket(s).

This works very fine for a socket.io application, which is running only on one server. Now, i want to configure another server with the same programs, modules and packages. The load balancing will probably be handled by HAProxy. So the socket.io connections (sockets) will be stored and managed on Server A and Server B.

Example scenario:

User A connects to Server A and User B connects to Server B. That means, that User A has a socket on Server A und User B on Server B.

How is it possible, that the application knows, that it has to look for the socket of User B on Server B to send the message? On Server A it won't find the socket, because it was created on Server B.

Thx a lot!

Answer 1

When you're horizontally scaling, you need to share a datastore between your servers. Conveniently, you already have an ideal one, which is Redis. Instead of keeping your socket mapping in an array, you need to push it into Redis, and do lookups from there.

Then, you can either decide to have servers send messages to each other, or, more scalably, send the messages through Redis as well. So, each server would look at it's queue on Redis to see what messages it should send to which sockets. When a server receives a message, it will push it into Redis addressed to the server that should deliver it. If the ordering of messages matters to you, I would recommend looking at https://github.com/learnboost/kue as layer on top of Redis.