Forcing new database connection when SSL SYSCALL error and CONN_MAX_AGE > 0

Question

Using Django 1.7 and Celery on Heroku with Postgres and RabbitMQ.

I recently set the CONN_MAX_AGE setting in Django to 60 or so so I could start pooling database connections. This worked fine until I discovered a problem where if for any reason a database connection was killed, Celery would continue using the bad database connection, consuming tasks but immediately throwing the following error within each task:

OperationalError: SSL SYSCALL error: Bad file descriptor

I would like to keep pooling database connections, but this has happened a few times now and I obviously can't allow Celery to randomly fail. How can I get Django (or Celery) to force a new database connection only when this error is hit?

(Alternatively, another idea I had was to force the Celery worker to run with a modified settings.py that sets CONN_MAX_AGE=0 only for Celery... but that feels very much like the wrong way to do it.)

Please note that this StackOverflow question seems to solve the problem on Rails, but I haven't found an equivalent for Django: On Heroku, Cedar, with Unicorn: Getting ActiveRecord::StatementInvalid: PGError: SSL SYSCALL error: EOF detected

Answer 1

I had the same problem and tracked it down to a combination of CONN_MAX_AGE and CELERYD_MAX_TASKS_PER_CHILD. At that point it became obvious that it must be something to do with Celery not closing connections properly when a worker is replaced and from that I found this bug report: https://github.com/celery/celery/issues/2453

Upgrading to Celery 3.1.18 seems to have solved the issue for me.