Correctly setting up Flask-SQLAlchemy for multiple celery workers and threads

Question

I'm struggling to make my Flask, SQLAlchemy (mysql) and Celery setup work properly when there are multiple celery workers with multiple threads involved that all query the same database.

The problem is that I cannot figure out how and where to apply required changes that give the flask application and each celery worker an isolated database object.

From my understanding, separate sessions are required to avoid nasty database errors such as incomplete transactions that block other database queries.

This is my current project structure

/flask_celery.py

from celery import Celery

def make_celery(app):
    celery = Celery(app.import_name, backend=app.config['CELERY_RESULT_BACKEND'],
                    broker=app.config['CELERY_BROKER_URL'])
    celery.conf.update(app.config)
    TaskBase = celery.Task
    class ContextTask(TaskBase):
        abstract = True
        def __call__(self, *args, **kwargs):
            with app.app_context():
                return TaskBase.__call__(self, *args, **kwargs)
    celery.Task = ContextTask
    return celery

/app.py

#!/usr/bin/env python

import config
from app import app

app.run(port=82,debug=True, host='0.0.0.0')
#app.run(debug=True)

app/__init.py__

from flask import Flask
from celery import Celery
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate
from flask_celery import make_celery

app = Flask(__name__)
app.config.from_object('config')
app.secret_key = app.config['SECRET_SESSION_KEY']

db = SQLAlchemy(app)
migrate = Migrate(app, db)

celery = make_celery(app)

Answer 1

Maybe give SQLALCHEMY_BINDS a chance. It's a guideline about how to bind multiple databases.

I afraid there are still extra moves you should made.

• I assume you have config.py to hold app configurations. Adding SQLALCHEMY_BINDS in Config class with values you prepared, which maybe several other databases' uri.
• Handling model classes in models.py if file exists.
• Managing your bind_key as argument somehow(sorry i don't make this detailed).
• Dealing with bind_key argument to the right celery work...

I hope this may help you a little. And please let me know if you work this out, so I could edit this answer for people who have similar cases.