Sharing a database between Twisted and Django

Question

I am developing a multiplayer gaming server that uses Django for the webserver (HTML frontend, user authentication, games available, leaderboard, etc.) and Twisted to handle connections between the players and the games and to interface with the games themselves. The gameserver, the webserver, and the database may run on different machines.

What is the "best" way to architect the shared database, in a manner that supports changes to the database schema going forward. Should I try incorporating Django's ORM in the Twisted framework and used deferreds to make it non-blocking? Should I be stuck creating and maintaining two separate databases schemas / interfaces, one in Django's model and the other using twisted.enterprise.row?

Similarly, with user authentication, should I utilize twisted's user authentication functionality, or try to include Django modules into the gameserver to handle user authentication on the game side?

Answer 1

First of all I'd identify why you need both Django and Twisted. Assuming you are comfortable with Twisted using twisted.web and auth will easily be sufficient and you'll be able to reuse your database layer for both the frontend and backend apps.

Alternatively you could look at it the other way, what is Twisted doing better as a game server? Are you hoping to support more players (more simultaneous connections) or something else? Consider that if you must use threaded within twisted to do blocking database access that you are most likely not going to be able to efficently/reliably support hundreds of simultaneous threads. Remember python has a Global Interpreter Lock so threads are not necessarily the best way to scale.

You should also consider why you are looking to use a SQL Database and an ORM. Does your game have data that is really best suited to being stored in an relational database? Perhaps it's worth examining something like MongoDB or another key-value or object database for storing game state. Many of these NoSQL stores have both blocking drivers for use in Django and non-blocking drivers for use in Twisted (txmongo for example).

That said, if you're dead set on using both Django and Twisted there are a few techniques for embedding blocking DB access into a non-blocking Twisted server.

1. adbapi (uses twisted thread pool)
2. Direct use of the twisted thread pool using reactor.deferToThread
3. The Storm ORM has a branch providing Twisted support (it handles deferToThread calls internally)
4. SAsync is a library that tries to make SQLAlchemy work in an Async way
5. Have twisted interact via RPC with a process that manages the blocking DB

So you should be able to manage the Django ORM objects yourself by importing them in twisted and being very careful making calls to reactor.deferToThread. There are many possible issues when working with these objects within twisted in that some ORM objects can issue SQL when accessing/setting a property, etc.

I realize this isn't necessarily the answer you were expecting but perhaps more detail about what you're hoping to accomplish and why you are choosing these specific technologies will allow folks to get you better answers.