Using Linux Virtual Server for load balancing of zones in MMO game

Question

I'm a developer of a MMO game and currently we're at my company facing some scalability issues which, I think, can be resolved with proper clustering of the game world.

I don't really want to reinvent the wheel that's why I think Linux Virtual Server could be a good choice especially with some Level 7 load balancing technique.

I'm currently looking at ktcpvs as a load balancing solution and wonder if it's a proper choice.

The main idea is to have a number of zones("locations" in terms of my game) running on dedicated servers. When a player decides to go to some specific location the load balancer decides which zone server will be actually serving the player(that's actually why I need a Level 7 load balancer)

What do you folks think about all said above?

Update: I posted the same question to LVS users mailing list http://marc.info/?l=linux-virtual-server&m=124976265209769&w=2

Update: I also started the similar topic on the gamedev.net forum http://www.gamedev.net/community/forums/topic.asp?topic_id=544386

Answer 1

In order to address your question we need to understand whether you need volume or response, but it is difficult to get both at the same time.

Layer 7 load balancing - is data based application level balancing, so the data content of the network packet needs to be routed to an end-point. You can achieve volume (more users) by implementing routing at the application level, service level or kernel level.

Scalability - I assume you are running out of memory, CPU resources and network bandwidth.

• Application level - your application logic receives an application packet and routes accordingly.

• Service level - your system framework (front end service of some kind) receives the packet and through a module - performs the routing (think of custom apache module, even network driver modules - like writing a network filter)

• Kernel level - Performs routing at network packet level.

The closer you move to the metal, the better your response will be. I suggest using dedicated linux server up-front to perform the routing - go native, not virtual. Use multiple or teamed network adapters for the WAN and a dedicated adapter for each end-point (one+ wan, one each for each connected app server)

If response time is important then you need a kernel/supervisor state solution, it will save you a few context switches but be aware that you need to limit hops at all costs and could better be served by fewer, larger machines and your scalability will always be limited. There is a risk in using KTCPVS, it is quite old and not actively updated. If you judge that it works for you great, otherwise consider writing something akin to a network filter as long as it runs in system state.

If volume is important but response time is secondary, implement a custom built high-speed socket switch built in C++ running in problem/user state. It is the easiest to maintain and will offer the best scalability.

You will need to build some prototypes to figure out what suits your needs best.

Final thoughts -

Before doing any of the above first ensure that you have optimized your game design. You may know most of this, I list it here for the benefit of all.

(a) messages should fit comfortably within one network packet, less than 1500 bytes for most home routers

(b) Try to fit the logic of the routing in your game client instead of your servers. A simple download of a small table with zones and IP addresses to a client will allow you to forego all of the above.

(c) Try to limit zone visibility by to the clients, they should know about their zones and adjacent zones only (if you implement the point b above)

Hope this helps, sorry I cannot be more specific regarding KTCPVS.

Answer 2

You haven't specified where the bottleneck is. Network Traffic? Disk IO? CPU Cycles?

Assuming you mean a layer 7 load balancer and don't have enough CPU power, I think LVS ist not the optimal choice. I have done Web Server load balancing with LVS, which works straightforward and isn't exactly complicated.

But I think load balancing an MMORP this way needs considerable amounts of additional code in LVS, it might be easier to do the load balancing with a multithreaded application distributed over some multicore server. But this isn't fully scalable, this only gets you to 16 cores without prohibitve cost increase.