Exclude soft deleted items in self referential relationship SQLAlchemy

Question

I currently have a self referential relationship on the Foo:

parent_id = DB.Column(DB.Integer, DB.ForeignKey('foo.id'))

parent = DB.relation(
    'Foo', 
    remote_side=[id], 
    backref=DB.backref(
        'children', 
        primaryjoin=('and_(foo.c.id==foo.c.parent_id, foo.c.is_deleted==False)')
    )
)

Now I am trying to exclude any children with is_deleted set as true. I'm pretty sure the problem is it is checking is_deleted against the parent, but I have no idea where to go from here.

How to modify the relationship so that children with is_deleted are not included in the result set?

Answer 1

I took a stab at answering this. My solution should work with SQLAlchemy>=0.8.

In effect nothing surprising is going on here, yet proper care has to be applied when using such patterns, as the state of the Sessions identity-map will not reflect the state of the DB all the time.

I used the post_update switch in the relationship to break the cyclical dependency which arises from this setup. For more information have a look at the SQLAlchemy documentation about this.

Warning: The fact that the Session does not always reflect the state of the DB may be a cause for nasty bugs and other confusions. In this example I use expire_all to show the real state of the DB, yet this is not a good solution because it reloads all objects and all un-flushed changes are lost. Use expire and expire_all with great care!

First we define the model

#!/usr/bin/env python
import sqlalchemy as sa
import sqlalchemy.orm as orm
from sqlalchemy.ext.declarative import declarative_base

engine = sa.create_engine('sqlite:///blah.db')
Base = declarative_base()
Base.bind = engine

class Obj(Base):
    __table__ = sa.Table(
        'objs', Base.metadata,
        sa.Column('id', sa.Integer, primary_key=True),
        sa.Column('parent_id', sa.Integer, sa.ForeignKey('objs.id')),
        sa.Column('deleted', sa.Boolean),
    )

    # I used the remote() annotation function to make the whole thing more
    # explicit and readable.
    children = orm.relationship(
        'Obj',
        primaryjoin=sa.and_(
            orm.remote(__table__.c.parent_id) == __table__.c.id,
            orm.remote(__table__.c.deleted) == False,
        ),
        backref=orm.backref('parent',
                            remote_side=[__table__.c.id]),
        # This breaks the cyclical dependency which arises from my setup.
        # For more information see: http://stackoverflow.com/a/18284518/15274
        post_update=True,
    )

    def __repr__(self):
        return "<Obj id=%d children=%d>" % (self.id, len(self.children))

Then we try it out

def main():
    session = orm.sessionmaker(bind=engine)
    db = session()
    Base.metadata.create_all(engine)

    p1 = Obj()
    db.add(p1)
    db.flush()

    p2 = Obj()
    p2.deleted = True

    p1.children.append(p2)
    db.flush()

    # prints <Obj id=1 children=1>
    # This means the object is in the `children` collection, even though
    # it is deleted. If you want to prevent this you may want to use
    # custom collection classes (not for novices!).
    print p1

    # We let SQLalchemy forget everything and fetch the state from the DB.
    db.expire_all()

    p3 = db.query(Obj).first()

    # prints <Obj id=1 children=0>
    # This indicates that the children which is still linked is not
    # loaded into the relationship, which is what we wanted.
    print p3

    db.rollback()


if __name__ == '__main__':
    main()