gevent and posgres: Asynchronous connection failed

Question

I'm using gevent to handle API I/O on a Django-based web system.

I've monkey-patched using:

import gevent.monkey; gevent.monkey.patch_socket()

I've patched psychopg using:

import psycogreen; psycogreen.gevent.patch_psycopg()

Nonetheless, certain Django calls so Model.save() are failing with the error: "Asynchronous Connection Failed." Do I need to do something else to make postgres greenlet-safe in the Django environment? Is there something else I'm missing?

Answer 1

there is an article on this promblem, unfortunately it's in Russian. Let me quote the final part:

All the connections are stored in django.db.connections, which is the instance of django.db.utils.ConnectionHandler. Every time ORM is about to issue a query, it requests a DB connection by calling connections['default']. In turn, ConnectionHandler.__getattr__ checks if there is a connection in ConnectionHandler._connections, and creates a new one if it is empty.

All opened connections should be closed after use. There is a signal request_finished, which is run by django.http.HttpResponseBase.close. Django closes DB connections at the very last moment, when nobody could use it anymore - and it seems reasonable.

Yet there is tricky part about how ConnectionHandler stores DB connections. It uses threading.local, which becomes gevent.local.local after monkeypatching. Declared once, this structure works just as it was unique at every greenlet. Controller *some_view* started its work in one greenlet, and now we've got a connection in *ConnectionHandler._connections*. Then we create few more greenlets and which get an empty *ConnectionHandlers._connections*, and they've got connectinos from pool. After new greenlets done, the content of their local() is gone, and DB connections gone withe them without being returned to pool. At some moment, pool becomes empty

Developing Django+gevent you should always keep that in mind and close the DB connection by calling django.db.close_connection. It should be called at exception as well, you can use a decorator for that, something like:

class autoclose(object):
    def __init__(self, f=None):
        self.f = f

    def __call__(self, *args, **kwargs):
        with self:
            return self.f(*args, **kwargs)

    def __enter__(self):
        pass

    def __exit__(self, exc_type, exc_info, tb):
        from django.db import close_connection
        close_connection()
        return exc_type is None