Django on GoogleAppEngine: performance howto

Question

I asked this question a few weeks ago. Today I have actually written and released a standard Django application, i.e. a fully-functional relational DB-backed (and consequently fully-functional Django admin) enabled by Google CloudSQL. The only time I had to deviate from doing things the standard Django way was to send email (had to do it the GAE way). My setup is GAE 1.6.4, Python2.7, Django 1.3 using the following in app.yaml:

libraries:
- name: django
  version: "1.3"

However I do need you to suggest clear actionable steps to improve to the response time of the initial request when cold of this Django app. I have a simple webapp2 web site on GAE, which does not hit the DB, and when cold the response time is of 1.56s. The Django one, when cold, hits the DB with 2 queries (two count(*) queries over tables containing less than 300 rows each), and the response time is of 10.73s! Not encouraging for a home page ;)

Things that come to mind are to remove the middleware classes I don't need and other Django-specific optimisations. However tips that improve things also from a GAE standpoint would be really useful.

N.B. I don't want this to become a discussion about the merits of going for Django on GAE. I can mention that my personal Django expertise, and resulting development productivity, did bear considerably in adopting Django as opposed to other frameworks. Moreover with CloudSQL, it's easy to move away from GAE (hopefully not!) as the Django code will work everywhere else with little (or no) modifications. Related discussions about such topic can be found here and here.

Answer 1

I don't have a full answer but I'm contributing since I'd like to find a solution as well. I'm currently using a running cron job (I actually need the cron job, so it's not only for keeping my app alive).

I've seen it discussed in one of the GAE/Python/Django related mailing lists that just the time required for loading up all the Django files is significant when compared to webapp, so removing django components that you don't use from deployment should improve your startup time as well. I've been able to shave off about 3 seconds by removing certain parts of the contrib folder. I exclude them in my app.yaml.

My startup time is still around 6 seconds (full app, Django-nonrel, HRD). It used to be more like 4 when my app was simpler.

My suspicion is that Django verifies all its models on startup, and that processing time is significant. If you have time with experimenting with an app with absolutely 0 models, I'd be curious if it made any impact.

I'm also curious as to whether your two initial queries make any significant impact.

Answer 2

When there is no instance running, for example after version upgrade or when there is no request for 15 min, then a request will trigger loading of an instance which takes about 10s. So what you are seeing is normal.

So if your app is idle for periods longer then 15min you will see this behavior. One workaround is to have a cron job ping your instance every 10 min (though I believe google does not like that).

Update:

You can avoid this by enabling billing then in GAE admin under "Application settings" set minimum Idle Instances setting to 1. Note: the min setting is not available on free apps, only max.