Signal/Method for every executed sql statement

Question

According to this 12 years old issue, django does not support a signal for every executed sql statement: https://code.djangoproject.com/ticket/5415

I need this in a production environment where debug=False.

This means overwriting connection.queries does not work.

Is there a way to run some custom code after each sql statement (even if debug=False)?

Answer 1

I've taken a look at how Django populates connection.queries. When DEBUG = True, the base database backend code that is used by all backend wraps the database-specific cursors with CursorDebugWrapper. Otherwise, it uses CursorWrapper. It would be in theory possible to force Django to populate it even when DEBUG = False by overriding the queries_logged property or setting the flag force_debug_cursor on the database connection object. Either method would force Django to use CursorDebugWrapper even when DEBUG is False. However, I don't recommend this approach because CursorDebugWrapper is not particularly efficient for cases where the only thing you want is to know that a query was executed. For instance, besides populating connection.queries it logs the queries to a logger. If you do not need this logging, then it is just a waste.

So taking inspiration from how CursorDebugWrapper works, I've come up with a custom way to know when a SQL query has been made.

I created an app called first whose __init__.py is:

from django.db.backends import utils
from contextlib import contextmanager

# This is inspired by Django's stock CursorDebugWrapper class.
class Mixin(object):

    def execute(self, sql, params=None):
        with self.notify(sql, params, use_last_executed_query=True):
            return super().execute(sql, params)

    def executemany(self, sql, param_list):
        with self.notify(sql, param_list):
            return super().executemany(sql, param_list)

    @contextmanager
    def notify(self, sql=None, params=None, use_last_executed_query=False):
        try:
            yield
        finally:
            if use_last_executed_query:
                sql = self.db.ops.last_executed_query(self.cursor, sql, params)
            # I've used print for this proof-of-concept, replace with whatever
            # mechanism suits you.
            print("Executed: ", sql)

class CustomWrapper(Mixin, utils.CursorWrapper):
    pass

class CustomDebugWrapper(Mixin, utils.CursorDebugWrapper):
    pass

utils.CursorWrapper = CustomWrapper
utils.CursorDebugWrapper = CustomDebugWrapper

Then in my settings.py file, I've put the first app first INSTALLED_APPS. This makes it so that the __init__.py file for it executes before anything that accesses the database.

What this is doing is essentially replacing the stock CursorWrapper and CursorDebugWrapper classes with custom ones that allow knowing when an SQL query has occurred. This change takes effect for all the backends that are shipped with Django.

I don't know of another way to add this capability in one fell swoop like I'm doing above. I first looked to see if I could derive a new backend from an existing one. I'm sure it is doable but it requires a lot of boilerplate. Also, projects that use multiple backends at once would have to derive a new backend for each stock backend that was originally used.

Answer 2

I do believe that the interesting solution posted by @Louis should be flagged as the the correct one.

Having said this, if your need is that of logging and analyzing all executed queries, you might consider installing django-debug-toolbar and configuring it to be optionally available in production for the superuser or a specific administration user.

For example, put this in myproject/settings.py:

def show_toolbar(request):
    from constance import config
    if not config.DEBUG_SHOW_TOOLBAR:
        return False
    return request.user.is_superuser


DEBUG_TOOLBAR_CONFIG = {
    'SHOW_TOOLBAR_CALLBACK': 'myproject.settings.show_toolbar',
    'INTERCEPT_REDIRECTS': False,
}

django-debug-toolbar has a specific panel with very detailed informations about all database activity.