Can you really scale up with Django...given that you can only use one database? (In the models.py and settings.py)

Question

Django only allows you to use one database in settings.py. Does that prevent you from scaling up? (millions of users)

Answer 1

Django now has support for multiple databases.

Answer 2

The database isn't your bottleneck.

Check your browser carefully.

For each page of HTML you're sending (on average) 8 other files, some of which may be quite large. These are your JS, CSS, graphics, etc.

The actual performance bottleneck is the browser requesting those files and accepting the bytes s... l... o... w... l... y...

To scale, then, do this.

1. Use multiple front-ends balanced with a pure software solution like wackamole. http://www.backhand.org/wackamole/

2. Use proxy servers like squid to send the "other" files. They're largely static. This is where 7/8ths of the work is done downloading to the client. Don't scrimp on getting these right.

3. Use multiple, concurrent mod_wsgi/Django to create the -- rare -- piece of dynamic HTML based on DB queries. Be sure that mod_wsgi is in daemon mode so that you can have multiple Django servers available to Apache. Build as many of these as you need. They're all identical, all in parallel, and all shared by Wackamole.

4. Use a single, fast database like MySQL for the few things that must come from a database. MySQL will make use of multiple cores on it's server, so it will scale reasonably well without you having to do anything other than buy memory. Put this on a separate box, all by itself, dedicated and tuned for just this.

You'll find that this scales nicely. You'll find that the load is shared nicely between squid, apache, the Django daemons and the actual database. You'll also find that each part of the load (from the boring static parts to the interesting database query) happens separately and concurrently.

Finally, buy Schlossnagle's book. http://www.amazon.com/Scalable-Internet-Architectures-Theo-Schlossnagle/dp/067232699X

Answer 3

Read scaling to millions of users is not a database problem, but is fixed with load balancing and caching, etc, see S. Lott above.

Write scaling can indeed be a database problem. "Sharding" and having multiple databases can be one solution, but that's hard with SQL while still retaining the relationality of the database. Popular solutions there are the new types of "nosql" databases. But if you really have those problems, then you need serious expert help, not just answers from dudes Stackoverflow. :)

Answer 4

Some great answers already (S. Lott for example), however I thought I should pipe in with some more things:

Make sure not to use the database for logical operations

I understand the attractiveness of Order By or SQL Procedures however you only have one database but you have multiple django servers, let the servers handle this if you can.

Of course, if you only want the last ten rows according to a certain criterion (date), then by all means do precise it in the request ;) Just make sure not to overload your database with operations that could be handled elsewhere.

Throw more hardware to the problem

MySQL and Oracle scale quite well with hardware, if you have a small problem of performance you could begin by adding more hardware.

Split your database

I know that for relationships and all you have to manage some tables together, however if you ever have a load problem, try to group your tables, for example if you have a "history" group of tables, perhaps that it could work without the others and be on a separate server.

Do consider tuning, and watch out for your requests/index

You would need experts advises here, but I can tell from experience that even a single badly tuned request can wreak havoc... and it's quite difficult to find out. You can consider the Ask Tom website for example of diagnosis / fine tuning.

Don't decide on your tables architecture in isolation, but do consider the requests

Hierarchical requests and multiple joins can be really costly. You don't have to build a fully normalized relations schema and may consider some denormalization in order to better accomodate the type of requests the database will face.

Just a couple of thoughts :)

Answer 5

A few miscellaneous pieces of advice:

• I'm surprised no one's mentioned this yet. Use memcached. If you're getting a lot of repetitive types of queries (which most webapps do), this can make a huge difference.

• Consider using Oracle's failover and load balancing. It allows you to add support for multiple databases on a single db connection.

• Another thing to consider is using a system similar to FriendFeed's. This solves the problem of "how do we make changes to the database without halting the world?" more than anything else.