Is there a production-safe way to measure time spent in Production w/Python?

Question

I want to be able to instrument Python applications so that I know:

• Page generation time.
• Percentage of time spent in external requests (mysql, api calls).
• Number of mysql queries, what the MySQL queries were.

I want this data from production (not offline profiling) - because the time spent in various places will be different under load.

In PHP I can do this with XHProf or instrumentation-for-php. In Ruby on Rails/.NET/Java, I can do this with New Relic.

Is there such a package recommended for Python or django?

Answer 1

Yes, it's perfectly possible. E.g. use some magic switch in URL, like "?profile-me" which triggers profiling in Django middleware.

There are a number of snippets on the Internet, like this one: http://djangosnippets.org/snippets/70/ or modules like this one: http://code.google.com/p/django-profiling/ - but I haven't used any of them so I cannot recommend anything.

Anyway, the approach they take is similar to what I do - i.e. use Python Hotshot profiler module in a middleware that wraps your view. For the MySQL part, you can just use connection.queries form Django.

The nice thing about Hotshot is that its output can be browsed using Kcachegrind like here: http://www.rkblog.rk.edu.pl/w/p/django-profiling-hotshot-and-kcachegrind/