Scaling WebSockets with a Message Queue

Question

I have built a WebSockets server that acts as a chat message router (i.e. receiving messages from clients and pushing them to other clients according to a client ID).

It is a requirement that the service be able to scale to handle many millions of concurrent open socket connections, and I wish to be able to horizontally scale the server.

The architecture I have had in mind is to put the websocket server nodes behind a load balancer, which will create a problem because clients connected to different nodes won't know about each other. While both clients A and B enter via the LoadBalancer, client A might have an open connection with node 1 while client B is connected to node 2 - each node holds it's own dictionary of open socket connections.

To solve this problem, I was thinking of using some MQ system like ZeroMQ or RabbitMQ. All of the websocket server nodes will be subscribers of the MQ server, and when a node gets a request to route a message to a client which is not in the local connections dictionary, it will pub-lish a message to the MQ server, which will tell all the sub-scriber nodes to look for this client and issue the message if it's connected to that node.

Q1: Does this architecture make sense?

Q2: Is the pub-sub pattern described here really what I am looking for?

Answer 1

To update this for 2021, we just solved this problem where we needed to design a system that could handle millions of simultaneous WS connections from IoT devices. The WS server just relays messages to our Serverless API backend that handles the actual logic. We chose to use docker and the node ws package using an auto-scaling AWS ECS Fargate cluster with an ALB in front of it.

This solved the main problem of routing messages, but then we had the same issue of how do we route response messages from the server. We initially thought of just keeping a central DB of connections, but routing messages to a specific Fargate instance behind an ALB didn't seem feasible.

Instead, we set up a simple sub/pub pattern using AWS SNS (https://aws.amazon.com/pub-sub-messaging/). Every WS server receives the response and then searches its own WS connections. Since each Fargate instance handles just routing (no logic), they can handle a lot of connections when we vertically scale them.