Is Project Darkstar Realistic? [closed]

Question

Project Darkstar was the topic of the monthly JavaSIG meeting down at the Google offices in NYC last night. For those that don't know (probably everyone), Project Darkstar is a framework for massively multiplayer online games that attempts to take care of all of the "hard stuff." The basic idea is that you write your game server logic in such a way that all operations are broken up into tiny tasks. You pass these tasks to the Project Darkstar framework which handles distributing them to a specific node in the cluster, any concurrency issues, and finally persisting the data.

Apparently doing this kind of thing is a much different problem for video games than it is for enterprise applications. Jim Waldo, who gave the lecture, claims that MMO games have a DB read/write ratio of 50/50, whereas enterprise apps are more like 90% read, 10% write. He also claims that most existing MMOs keep everything in memory exlcusively, and only dump to a DB every 6 hours of so. This means if a server goes down, you would lose all of the work since the last DB dump.

Now, the project itself sounds really cool, but I don't think the industry will accept it. First, you have to write your server code in Java. The client code can be written in anything (Jim claims ActionScript 3 is the most popular, follow by C++), but the server stuff has to be Java. Sounds good to me, but I really get the impression that everyone in the games industry hates Java.

Second, unlike other industries where developers prefer to use existing frameworks and libraries, the guys in the games industry seem to like to write everything themselves. Not only that, they like to rewrite everything for every new game they produce. Things are starting to change where developers are using Havok for physics, Unreal Engine 3 as their platform, etc., but for the most part it looks like everything is still proprietary.

So, are the guys at Project Darkstar just wasting their time? Can a general framework like this really work for complex games with the performance that is required? Even if it does work, are game companies willing to use it?

Answer 1

Edit: This was written before Oracle bought Sun and started a rampage to kill everything that does not make them a billion $ per day. See the comments for an OSS Fork. I still stand by my opinion that stuff like that (MMO Middleware) is realistic, you just need a company that doesn't suck behind it.

The Market may be dominated by few large games, but that does not mean that there is not a lot of room for more niche games. Lets face it: If you want to reach 100.000+ players, you're ending up building your own technology stack, at least for the critical core. That's what CCP did for EVE Online (StacklessIO), that's what Blizzard did for World of Warcraft (although they do use many third-party libraries), that's what Mythic did for Warhammer Online (although they are based on Gamebryo).

However, if you aim to be a small, niche MMO (like the dozens of Free-to-Play/Itemshop MMOs), then getting the Network stuff right is just insanely hard, data consistency is even harder and scalability is the biggest b*tch.

But game technology is not your only problem - you also need to tackle Billing. Credit Card only? Have fun selling in Germany then, people there want ELV. That's where you need a reliable billing provider, but you still need to wire in the billing application with your accounts to make sure that accounts are blocked/reactivated when the billing fails.

There are some companies already offering "MMO Infratructure Services" (i.e. Arvato's EEIS), but the bottom line is: Stuff like Project Darkstar IS realistic, but assuming that you can build a Multi-Billion-MMO entirely on a Third Party Stack is optimistic, possibly idealistic.

But then again, entirely inventing all of the technology is even more stupid - use the Third Party stuff that you need (i.e. Billing, Font Rendering, Audio Output...), but write the stuff that really makes or breaks your business (i.e. Network stack, User interface etc.) on your own. (Note: Jeff's posting may be a bit flawed, but the overall direction is correct IMHO.)

Addendum: Also, the game industry does license and reuse engines a lot. The most prominent game Engines are the Unreal Engine, Source Engine and id Tech, which fuel dozens, if not hundreds of games. But there are some lesser-known (outside of the industry) engines. There is Gamebryo, the Middleware behind games like Civilization 4 and Fallout 3, there was RenderWare that is now only EA-in-House, but used in games like Battlefield 2 or The Sims 3. There is the open source Ogre3d, which was used in some commercial titles. If you're just looking for Sound, there's stuff like FMOD or if you want to do font-rendering, why not give FreeType a spin?

What I'm saying is: Third-Party Engines/Middleware do exist, and they ARE being successfully used since more than a decade (I know for sure that id's Wolfenstein Engine was licensed to other companies, and that was 1992), even by big companies in multi-million-dollar titles. The important thing is the support, because a good engine with no help in case of an issue is pretty much worthless or at least very expensive if the developer has to spend their game-development-time with unneccessary debugging of the Engine.

If the Darkstar folks manage to get the support side right and 2 or 3 higher profile titles out, I do believe it could succeed in opening the MMO market to a lot more smaller developers and indies.