Celery, Redis and ConnectionPool

Question

I recently had a problem with Celery and Redis : I was using too many connections to my cloud Redis account.

I seem to have fixed the problem for now with better settings and a bigger Redis plan.

However, it lead me to some experimentation on how Redis and Celery work together, and there is something I don't understand : the number of Redis ConnectionPool that are created by the Python Redis module.

I configured Celery to use Redis as a broker and a result backend.

Here is my Celery config :

CELERY_TIMEZONE = 'Europe/Paris' CELERY_BROKER_URL = REDIS_URL CELERY_RESULT_BACKEND = REDIS_URL CELERY_TASK_SERIALIZER = 'pickle' CELERY_SEND_EVENTS = False CELERY_IMPORTS = ('task1', 'task2') CELERY_ACCEPT_CONTENT = ['pickle'] CELERY_REDIS_MAX_CONNECTIONS = 2 BROKER_POOL_LIMIT = 1 BROKER_HEARTBEAT = None BROKER_TRANSPORT_OPTIONS = {    'max_connections': 30 } 

I run Celery with the following command :

celery worker --without-gossip --without-mingle --without-heartbeat --concurrency=1 --app=backZest.celery

Right after Celery has booted, I already have 10 connections to my Redis instance : is that normal?

I checked how many redis ConnectionPool were created : to do this, I just modified the Python Redis module connection.py file, in the __init__ method of the ConnectionPool class:

import sys print "ConnectionPool instance %s with %s max_connections" % (id(self), self.max_connections) sys.stdout.flush() 

As you can see, there are many (8) pools created, all of them with the same max_connections, 30, which is my setting in BROKER_TRANSPORT_OPTIONS :

ConnectionPool instance 4412957392 with 30 max_connections ConnectionPool instance 4412959888 with 30 max_connections ConnectionPool instance 4420369616 with 30 max_connections ConnectionPool instance 4420370320 with 30 max_connections ConnectionPool instance 4420367056 with 30 max_connections ConnectionPool instance 4420491792 with 30 max_connections ConnectionPool instance 4422318224 with 30 max_connections ConnectionPool instance 4422319504 with 30 max_connections 

I don't understand why there are so many of them. I want to control the number of connections to my Redis. I can control the number of max_connections per ConnectionPool but isn't it a bit useless if I don't control the number of ConnectionPool created ?

Thanks a lot for your help !

Answer 1

Yes, this is normal. Pool creates connections beforehand so they are ready when you need them. Establishing a connection may take a lot of time, so pool helps you save on that. Use max_connections to restrict the pool as you need it. And don't worry that pool opens connections early.