django-south with django-audit-log

Question

I'm trying to do a django-south migration to an existing application to add django-audit-log to it (to track user-initiated changes of a module), but am running into significant errors. Specifically with the action_user_id field that is a LastUserField (which stores the user who specified the change that is being tracked).

If I was starting from a blank model, I could just add an audit_log via:

from audit_log.models.managers import AuditLog
...
class SomeModel(models.Model)
    ...
    audit_log = AuditLog()

Applying this simple change and doing a schemamigration in django-south understandingly gives me an error:

 ! Cannot freeze field 'myapp.mymodelauditlogentry.action_user'
 ! (this field has class audit_log.models.fields.LastUserField)

 ! South cannot introspect some fields; this is probably because they are custom
 ! fields. If they worked in 0.6 or below, this is because we have removed the
 ! models parser (it often broke things).
 ! To fix this, read http://south.aeracode.org/wiki/MyFieldsDontWork

I read the MyFieldsDontWork wiki (and the Custom Fields/Introspection parts), but its not 100% clear what I need to do to get the fields to work.

I try adding:

from south.modelsinspector import add_introspection_rules
add_introspection_rules([], ["^audit_log\.models\.fields\.LastUserField"])

to my models.py which allowed the ./manage.py schemamigration to create a migration script with the previous error goes away. However when I try to migrate (to apply the migration), I get the following errors:

Running migrations for myapp:
 - Migrating forwards to 0004_auto__add_mymodelauditlogentry.
 > my_app:0004_auto__add_mymodelauditlogentry
Traceback (most recent call last):
  File "./manage.py", line 11, in <module>
    execute_manager(settings)
      File "/usr/local/lib/python2.6/dist-packages/Django-1.2.3-py2.6.egg/django/core/management/__init__.py", line 438, in execute_manager
    utility.execute()
  File "/usr/local/lib/python2.6/dist-packages/Django-1.2.3-py2.6.egg/django/core/management/__init__.py", line 379, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python2.6/dist-packages/Django-1.2.3-py2.6.egg/django/core/management/base.py", line 191, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/local/lib/python2.6/dist-packages/Django-1.2.3-py2.6.egg/django/core/management/base.py", line 220, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.6/dist-packages/South-0.7.3-py2.6.egg/south/management/commands/migrate.py", line 105, in handle
    ignore_ghosts = ignore_ghosts,
  File "/usr/local/lib/python2.6/dist-packages/South-0.7.3-py2.6.egg/south/migration/__init__.py", line 191, in migrate_app
    success = migrator.migrate_many(target, workplan, database)
  File "/usr/local/lib/python2.6/dist-packages/South-0.7.3-py2.6.egg/south/migration/migrators.py", line 221, in migrate_many
    result = migrator.__class__.migrate_many(migrator, target, migrations, database)
  File "/usr/local/lib/python2.6/dist-packages/South-0.7.3-py2.6.egg/south/migration/migrators.py", line 292, in migrate_many
    result = self.migrate(migration, database)
  File "/usr/local/lib/python2.6/dist-packages/South-0.7.3-py2.6.egg/south/migration/migrators.py", line 125, in migrate
    result = self.run(migration)
  File "/usr/local/lib/python2.6/dist-packages/South-0.7.3-py2.6.egg/south/migration/migrators.py", line 93, in run
    south.db.db.current_orm = self.orm(migration)
  File "/usr/local/lib/python2.6/dist-packages/South-0.7.3-py2.6.egg/south/migration/migrators.py", line 246, in orm
    return migration.orm()
  File "/usr/local/lib/python2.6/dist-packages/South-0.7.3-py2.6.egg/south/utils.py", line 62, in method
    value = function(self)
  File "/usr/local/lib/python2.6/dist-packages/South-0.7.3-py2.6.egg/south/migration/base.py", line 422, in orm
    return FakeORM(self.migration_class(), self.app_label())
  File "/usr/local/lib/python2.6/dist-packages/South-0.7.3-py2.6.egg/south/orm.py", line 46, in FakeORM
    _orm_cache[args] = _FakeORM(*args)  
  File "/usr/local/lib/python2.6/dist-packages/South-0.7.3-py2.6.egg/south/orm.py", line 125, in __init__
    self.models[name] = self.make_model(app_label, model_name, data)
  File "/usr/local/lib/python2.6/dist-packages/South-0.7.3-py2.6.egg/south/orm.py", line 318, in make_model
    field = self.eval_in_context(code, app, extra_imports)
  File "/usr/local/lib/python2.6/dist-packages/South-0.7.3-py2.6.egg/south/orm.py", line 236, in eval_in_context
    return eval(code, globals(), fake_locals)
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.6/dist-packages/django_audit_log-0.2.1-py2.6.egg/audit_log/models/fields.py", line 12, in __init__
    super(LastUserField, self).__init__(User, null = True, **kwargs)
TypeError: __init__() got multiple values for keyword argument 'null'

EDIT (12/20 noon): I can apply the schemamigration if I add the lines to models.py

from south.modelsinspector import add_introspection_rules, add_ignored_fields
add_ignored_fields(["^audit_log\.models\.fields\.LastUserField"])

except then the audit_log middleware doesn't work as there is no action_user_id integer field in myapp_mymodelauditlogentry that references "auth_user" by "id". Then I manually apply the SQL (sqlite syntax; obtained by using sqliteman on newly created database.)

ALTER TABLE "myapp_mymodelauditlogentry" ADD "action_user_id" integer REFERENCES "auth_user" ("id");

and it works. I'll still give the bounty if someone explains how I'm supposed to do this in the context of django-south with migrations/introspection, without necessitating going to raw database dependent SQL and be grateful.

Also, I created an index for action_user_id. I notice that the normal creation of models with leads to an index called

CREATE INDEX "myapp_mymodelauditlogentry_26679921" ON "myapp_mymodelauditlogentry" ("action_user_id")

I hunted down that the hash 26679921 is created based on the field name with '%x' % (abs(hash(('action_user_id',))) % 4294967296L,) and isn't based on anything else (so should always be _26679921 unless the database requires the long name to be trunctated). I'm not sure if the names of the index ever matter; but wanted to be safe.

Answer 1

Here's finally the answer (and explanation).

When migrating South not only stores the names of the fields in your models, but also the type and the arguments that are passed to it. The result of this is that South has to understand which parameters are given by the field and which should be stored.

So when you create a rule like this:

add_introspection_rules([], ["^audit_log\.models\.fields\.LastUserField"])

Than South will create a table with a column like this:

(
  'action_user',
  self.gf('audit_log.models.fields.LastUserField')(
    related_name='_somemodel_audit_log_entry',
    null=True,
    to=orm['auth.User'],
  )
),

Which has, as you can see, a related_name parameter, a null parameter and a to parameter. Now let's take a look at the field definition:

class LastUserField(models.ForeignKey):                                      
    """                                                                      
    A field that keeps the last user that saved an instance                  
    of a model. None will be the value for AnonymousUser.                    
    """                                                                      

    def __init__(self, **kwargs):                                            
        models.ForeignKey.__init__(self, User, null=True, **kwargs)          
        #print kwargs                                                        
        #super(LastUserField, self).__init__(User, null = True, **kwargs)    

    def contribute_to_class(self, cls, name):                                
        super(LastUserField, self).contribute_to_class(cls, name)            
        registry = registration.FieldRegistry(self.__class__)                
        registry.add_field(cls, self)                                        

What do we see here? The first argument to ForeignKey is user (the first argument is the to attribute). The second argument (also hardcoded) is the null parameter. The result, when applying the migration both South and your field will try to set these parameters.

And you get the error:

TypeError: __init__() got multiple values for keyword argument 'null'

How do we fix this?

Well, we can tell South that we are passing these arguments as defaults so it can safely ignore them.

So we create a set of rules like this:

rules = [(                                          
    (fields.LastUserField,),                        
    [],                                             
    {                                               
        'to': ['rel.to', {'default': User}],        
        'null': ['null', {'default': True}],        
    },                                              
)]   
add_introspection_rules(                           
    rules,                                         
    ['^audit_log\.models\.fields\.LastUserField'], 
)       

Because of that, South now understands how to store the parameters and which parameters need to be ignored. So the new field definition will be this:

(
  'action_user',
  self.gf('audit_log.models.fields.LastUserField')(
    related_name='_somemodel_audit_log_entry'
  )
),

As we can see, the related_name is still here, but the to and null parameters have disappeared. So now we can safely apply the migration without getting conflicts.

Answer 2

Despite using the steps in @WoLpH's answer, I still couldn't create the migration. I had to modify the audit_log/models/fields.py file. Here's how my LastUserField field looks like:

class LastUserField(models.ForeignKey):
    """ 
    A field that keeps the last user that saved an instance
    of a model. None will be the value for AnonymousUser.
    """

    def __init__(self, **kwargs):
        kwargs.pop('null', None)
        kwargs.pop('to', None)
        super(LastUserField, self).__init__(User, null = True, **kwargs)

    def contribute_to_class(self, cls, name):
        super(LastUserField, self).contribute_to_class(cls, name)
        registry = registration.FieldRegistry(self.__class__)
        registry.add_field(cls, self)

The following was added to my models.py file (which didn't work) before I had to resort to doing this:

rules = [((fields.LastUserField,),
    [],    
    {   
        'to': ['rel.to', {'default': User}],
        'null': ['null', {'default': True}],
    },)]

# Add the rules for the `LastUserField`
add_introspection_rules(rules, ['^audit_log\.models\.fields\.LastUserField'])

Any suggestions on what I could do to avoid this hackery?