Recommended way to find the source of a query when using Django?

Question

I'm having difficulties with locating the source of the queries which I see in my database logs. I'm using Django, so the actual queries are generated automatically which makes a simple "grep" ineffective.

I'm thinking of patching the database cursors to append the current stacktrace to the query, something like:

for conn in connections.all():
    with conn.cursor() as c:
        ctype = type(c.cursor)
        orig_execute = ctype.execute
        def _patchedExecute(self, query, params=None):
            query = query + ' -- ' + traceback.format_stack()
            return orig_execute(self, query, params)
        ctype.execute = _patchedExecute
        orig_execute_many = ctype.executemany
        def _patchedExecuteMany(self, query, params=None):
            query = query + ' -- ' + traceback.format_stack()
            return orig_execute_many(self, query, params)
        ctype.executemany = _patchedExecuteMany

What I'm wondering:

• is there a ready made solution for this? (seems like a somewhat common problem to have)
• if not, are there better solutions than the one presented above? (I'm not all that familiar with the Django internals)

---

Edit: thank you for everyone who suggested Django Debug Toolbar. I'm already aware of that (and similar tools like Django Silk) and already using it, however in this particular case I'm more interested in the reverse problem. That is: starting from the database where I see that a particular kind of query is executed frequently or that it takes a long time and I would like to know where in my Django application am I generating those queries?

Answer 1

EDIT: added another option...

The code you are describing is exactly what https://pypi.org/project/django-sql-stacktrace/ should do for you. Simply pip install it and add it to your apps. The settings are trivial, but for completeness:

INSTALLED_APPS = (
    'sqlstacktrace',
)

SQL_STACKTRACE = True

That's it! However, that project looks a little unloved, so you might want a more popular and more recently maintained project...

A further option that should work straight out of the box for you is https://github.com/dobarkod/django-queryinspect. This doesn't allow you to go directly from SQL server logs, but should provide the necessary information in your web server logs instead.

There's a fair amount of configuration required here, but I've tried to summarize below for a system to log duplicate requests and ones that take longer than 100ms:

MIDDLEWARE_CLASSES += (
    'qinspect.middleware.QueryInspectMiddleware',
)
DEBUG = True
LOGGING = {
    ...
    'handlers': {
        ...
        'console': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
        },
        ...
    },

    'loggers': {
        ...
        'qinspect': {
            'handlers': ['console'],
            'level': 'DEBUG',
            'propagate': True,
        },
    },
    ...
}
QUERY_INSPECT_ENABLED = True
QUERY_INSPECT_LOG_QUERIES = True
QUERY_INSPECT_LOG_TRACEBACKS = True
QUERY_INSPECT_ABSOLUTE_LIMIT = 100 # in milliseconds

I should caveat that I've not used either myself yet, but they will be top of the list next time I hit such an issue in my own projects.