Handling multiple models.py with alembic

Question

Our web app is based on sqlalchemy in pyramid framework and we are looking to use alembic for managing database migrations. The web application consists of various packages that operate on one database. This consequently means we have multiple models.py that need to be migrated. I am confused as to how to handle this. I could progress some far using the following in my env.py

from pkg_a.app.models import Base as pkg_a_base
from pkg_b.app.models import Base as pkg_b_base
from pkg_c.app.models import Base as pkg_c_base

def combine_metadata(*args):
    m = MetaData()
    for metadata in args:
        for t in metadata.tables.values():
            t.tometadata(m)
    return m
target_metadata = combine_metadata(pkg_a_base, pkg_b_base, pkg_c_base)

This works great the first time. However, if I add one more model later, just adding that to this list doesn't do much. I was expecting that running

alembic revision -m "added a new model pkg_d.models" --version-path=migrations/versions --autogenerate

would generate a new version file that would have the code for adding the tables from pkg_d.models. But it isn't so. What am I doing wrong here.

Answer 1

If your packages are completely independent and separate then each of them should have a separate migration history - either stored inside each package (pkg_a.migrations, pkg_b.migrations etc.) or at least stored in a separate top-level migrations directory via having a separate section in alembic.ini and using -n parameter to alembic command to specify which section to use:

[pkg_a]
# path to migration scripts
script_location = migrations_a
sqlalchemy.url = xxx

[pkg_b]
script_location = migrations_b
sqlalchemy.url = xxx

[pkg_c]
script_location = migrations_c
sqlalchemy.url = xxx

And then you'll be able to use alembic revision -n pkg_a -m "added a new model pkg_a.models"

If, however, your models are dependent in any way then they should use a common Base - you do realize you don't have to keep all your SQLAlchemy stuff in a single models.py file, don't you? I would create a separate "base" package which would contain a common MetaData, Base and other SQLAlchemy configuration stuff which would then be imported by your other packages.