Inter-child-process communication options in node.js cluster

Question

So I'm working on a node.js game server application at the moment, and I've hit a bit of a wall here. My issue is that I'm using socket.io to accept inbound connections from game clients. These clients might be connected to one of several Zones or areas of the game world.

The basic architecture is displayed below. The master process forks a child process for each Zone of the game which runs the Zone Manager process; a process dedicated to maintaining the Zone data (3d models, positions of players/entities, etc). The master process then forks multiple "Communication Threads" for each Zone Manager it creates. These threads create an instance of socket.io and listen on the port for that Zone (multiple threads listening on a single port). These threads will handle the majority of the game logic in their own process as well as communicate with the database backing the game server. The only issue is that in some circumstances they may need to communicate with the Zone Manager to receive information about the Zone, players, etc.

As an example: A player wants to buy/sell/trade with a non-player character (NPC) in the Zone. The Zone Communication thread needs to ask the Zone Manager thread if the player is close enough to the NPC to make the trade before it allows the trade to take place.

The issue I'm running into here is that I was planning to make use of the node.js cluster functionality and use the send() and on() methods of the processes to handle passing messages back and forth. That would be fine except for one caveat I've run into with it. Since all child processes spun off with cluster.fork() can only communicate with the "master" process. The node.js root process becomes a bottleneck for all communication. I ran some benchmarks on my system using a script that literally just bounced a message back and forth using the cluster's Inter-process Communication (IPC) and kept track of how many relays per second were being carried out. It seems that eventually node caps out at about 20k per second in terms of how many IPC's it can relay. This number was consistent on both a Phenom II 1.8ghz quad core laptop, and an FX-8350 4.0ghz 8-core desktop.

Now that sounds pretty decently high, except that this basically means that regardless of how many Zones or Communication Threads there are, all IPC is still bottlenecking through a single process that acts as a "relay" for the entire application. Which means that although it seems each individual thread can relay > 20k IPCs per second, the entire application as a whole will never relay more than that even if it were on some insane 32 core system since all the communication goes through a single thread.

So that's the problem I'm having. Now the dilemma. I've read a lot about the various other options out there and read like 20 different questions here on stack about this topic and I've seen a couple things popping up regularly:

Redis: I'm actually running Redis on my server at the moment and using it as the socket.io datastore so that socket.io in multiple threads can share connection data so that a user can connect to any of N number of socket.io threads for their Zone so the server can sort of automatically load balance the incoming connections.

My concern with this is that it runs through the network stack. Hardly ideal for communication between multiple processes on the same server. I feel like the latency would be a major issue in the long run.

0MQ (zeromq/zmq): I've never used this one for anything before, but I've been hearing a bit about it lately. Based on the reading I've done, I've found a lot of examples of people using it with TCP sockets, but there's not a lot of buzz about people using it for IPC. I was hoping perhaps someone here had worked with 0MQ for IPC before (possibly even in node.js?) and could shine some light on this option for me.

dnode: Again I've never used this, but from what I've seen it looks like it's another option that is designed to work over TCP which means the network stack gets in the way again.

node-udpcomm: Someone linked this in another question on here (which I can't seem to find again unfortunately). I've never even heard of it, and it looks like a very small solution that opens up and listens on UDP connections. Although this would probably still be faster than TCP options, we still have the network stack in the way right? I'm definitely like about a mile outside of my "programmer zone" as is here and well into the realm of networking/computer architecture stuff that I don't know much about lol

Anyway the bottom line is that I'm completely stuck here and have no idea what the best option would be for IPC in this scenario. I'm assuming at the moment that 0MQ is the best option of the ones I've listed above since it's the only one that seems to offer an "IPC" option for communication protocol which I presume means it's using a UNIX socket or something that's not going through the network stack, but I can't confirm that or anything.

I guess I'm just hoping some people here might know enough to point me in the right direction or tell me I'm already headed there. The project I'm working on is a multiplayer game server designed to work "out of the box" with a multiplayer game client both powering their 3D graphics/calculations with Three.js. The client and server will be made open source to everyone once I get them all working to my satisfaction, and I want to make sure that the architecture is as scalable as possible so people don't build a game on this and then go to scale up and end up hitting a wall.

Anyway thanks for your time if you actually read all this :)

Answer 1

I think that 0MQ would be a very good choice, but I admit that I don't know the others :D

For 0MQ it's transparent what transport you decide to use, the library calls are the same. It's just about choosing particular endpoint (and thus transport) during the call to zmq_bind and zmq_connect at the beginning. There are basically 4 paths you may decide to take:

1. "inproc://<id>" - in-process communication endpoint between threads via memory
2. "ipc://<filepath>" - system-dependent inter-process communication endpoint
3. "tcp://<ip-address>" - clear
4. "pgm://..." or "epgm://..." - an endpoint for Pragmatic Reliable Multicast

So to put is simply, the higher in the list you are, the faster it is and the less problems considering latency and reliability you have to face. So you should try to keep as high as possible. Since your components are processes, you should go with the IPC transport, naturally. If you later on need to change something, you can just change the endpoint definition and you are fine.

Now what is in fact more important than the transport you choose is the socket type, or rather pattern you decide to use. You situation is a typical request-response kind of communication, so you can either do

1. Manager: REP socket, Threads: REQ socket; or
2. Manager: ROUTER, Threads: REQ or DEALER

The Threads would then connect their sockets to the single Manager socket assigned to them and that's all, they can start to send their requests and wait for responses. All they have to decide on is the path they use as the endpoint.

But to describe in details what all those socket types and patterns mean is definitely out of the scope of this post, but you can and should read more about it in the ZeroMQ Guide. There you can not only learn about all the socket types, but also about many different ways how to connect your components and let them talk to each other. The one I mentioned is just a very simple one. Once you understand, you can build arbitrary hierarchies. It's like LEGO ;-)

Hope it helped a bit, cheers!