Configure celery to wait for backend service to start

Question

When starting up celery, it retries connecting to my rabbitmq broker, which gives it the necessary time to load. This is good, because I'm using docker and I can't guarantee the order in which services start and precisely which service will be up when.

However, while trying to connect to the local mysql server I set up as the results backend, celery does not use the same measure of mercy and dies instantly, complaining, reasonably, that it can't lock the mysqld socket:

OperationalError: (2002, "Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)")

I would like to configure celery to retry several times before giving up, is that possible?

Thanks!

Answer 1

celery use sqlalchemy behind the scene, it does not ship with connect retry feature out of the box, however, you could adjust the connect timeout, to wait longer for mysql server, by default this value is only 10s, larger value helps.

assuming you are using pymysql/mysqldb as DB driver, it accepts a connect_timeout option, to specify this option from celery, you need set database_engine_options, which will be passed to the create_engine function of sqlalchemy, and set connect_args, which will be passed directly from sqlalchemy to DB driver, eg:

app.conf.database_engine_options = {'connect_args': {'connect_timeout': 600}}

another option is to use a custom connection creator function, manage the connection creation all by yourself, you could retry whatever times you want.

Answer 2

You can try depends_on option in docker-compose. It defines the order in which the services you want start. You can find the official documentation here.