Flask + SQLAlchemy - custom metaclass to modify column setters (dynamic hybrid_property)

Question

I have an existing, working Flask app that uses SQLAlchemy. Several of the models/tables in this app have columns that store raw HTML, and I'd like to inject a function on a column's setter so that the incoming raw html gets 'cleansed'. I want to do this in the model so I don't have to sprinkle "clean this data" all through the form or route code.

I can currently already do this like so:

from application import db, clean_the_data
from sqlalchemy.ext.hybrid import hybrid_property
class Example(db.Model):
  __tablename__ = 'example'

  normal_column = db.Column(db.Integer,
                            primary_key=True,
                            autoincrement=True)

  _html_column = db.Column('html_column', db.Text,
                           nullable=False)

  @hybrid_property
  def html_column(self):
    return self._html_column

  @html_column.setter
  def html_column(self, value):
    self._html_column = clean_the_data(value)

This works like a charm - except for the model definition the _html_column name is never seen, the cleaner function is called, and the cleaned data is used. Hooray.

I could of course stop there and just eat the ugly handling of the columns, but why do that when you can mess with metaclasses?

Note: the following all assumes that 'application' is the main Flask module, and that it contains two children: 'db' - the SQLAlchemy handle and 'clean_the_data', the function to clean up the incoming HTML.

So, I went about trying to make a new base Model class that spotted a column that needs cleaning when the class is being created, and juggled things around automatically, so that instead of the above code, you could do something like this:

from application import db
class Example(db.Model):
  __tablename__ = 'example'
  __html_columns__ = ['html_column'] # Our oh-so-subtle hint

  normal_column = db.Column(db.Integer,
                            primary_key=True,
                            autoincrement=True)

  html_column = db.Column(db.Text,
                          nullable=False)

Of course, the combination of trickery with metaclasses going on behind the scenes with SQLAlchemy and Flask made this less than straight-forward (and is also why the nearly matching question "Custom metaclass to create hybrid properties in SQLAlchemy" doesn't quite help - Flask gets in the way too). I've almost gotten there with the following in application/models/__init__.py:

from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.ext.hybrid import hybrid_property
# Yes, I'm importing _X stuff...I tried other ways to avoid this
# but to no avail
from flask_sqlalchemy import (Model as BaseModel,
                              _BoundDeclarativeMeta,
                              _QueryProperty)
from application import db, clean_the_data

class _HTMLBoundDeclarativeMeta(_BoundDeclarativeMeta):
  def __new__(cls, name, bases, d):
    # Move any fields named in __html_columns__ to a
    # _field/field pair with a hybrid_property
    if '__html_columns__' in d:
      for field in d['__html_columns__']:
        if field not in d:
          continue
        hidden = '_' + field
        fget = lambda self: getattr(self, hidden)
        fset = lambda self, value: setattr(self, hidden,
                                           clean_the_data(value))
        d[hidden] = d[field] # clobber...
        d[hidden].name = field # So we don't have to explicitly
                               # name the column. Should probably
                               # force a quote on the name too
        d[field] = hybrid_property(fget, fset)
      del d['__html_columns__'] # Not needed any more
    return _BoundDeclarativeMeta.__new__(cls, name, bases, d)

# The following copied from how flask_sqlalchemy creates it's Model
Model = declarative_base(cls=BaseModel, name='Model',
                         metaclass=_HTMLBoundDeclarativeMeta)
Model.query = _QueryProperty(db)

# Need to replace the original Model in flask_sqlalchemy, otherwise it
# uses the old one, while you use the new one, and tables aren't
# shared between them
db.Model = Model

Once that's set, your model class can look like:

from application import db
from application.models import Model

class Example(Model): # Or db.Model really, since it's been replaced
  __tablename__ = 'example'
  __html_columns__ = ['html_column'] # Our oh-so-subtle hint

  normal_column = db.Column(db.Integer,
                            primary_key=True,
                            autoincrement=True)

  html_column = db.Column(db.Text,
                          nullable=False)

This almost works, in that there's no errors, data is read and saved correctly, etc. Except the setter for the hybrid_property is never called. The getter is (I've confirmed with print statements in both), but the setter is ignored totally and the cleaner function is thus never called. The data is set though - changes are made quite happily with the un-cleaned data.

Obviously I've not quite completely emulated the static version of the code in my dynamic version, but I honestly have no idea where the issue is. As far as I can see, the hybrid_property should be registering the setter just like it has the getter, but it's just not. In the static version, the setter is registered and used just fine.

Any ideas on how to get that final step working?

Answer 1

Maybe use a custom type ?

from sqlalchemy import TypeDecorator, Text

class CleanedHtml(TypeDecorator):
    impl = Text

    def process_bind_param(self, value, dialect):
        return clean_the_data(value)

Then you can just write your models this way:

class Example(db.Model):
    __tablename__ = 'example'
    normal_column = db.Column(db.Integer, primary_key=True, autoincrement=True)
    html_column = db.Column(CleanedHtml)

More explanations are available in the documentation here: http://docs.sqlalchemy.org/en/latest/core/custom_types.html#augmenting-existing-types