Using __new__ on classes derived from Django's models does not work

Question

This is something which is puzzling me, but I cannot get a definitive answer. Using the __new__ method (or more accurately, static method) within classes derived from DJango model.

This is how __new__ should be ideally used (since we are using Django, we can assume that version 2.x of python is being used):

class A(object):
  def __new__(self, *args, **kwargs):
    print ("This is A's new function")
    return super(A, self).__new__(self, *args, **kwargs)

  def __init__(self):
    print ("This is A's init function")

Instantiating an object from the above class works as expected. Now, when one tries something like this on classes derived from Django models, something unexpected happens:

class Test(models.Model):
  def __new__(self, *args, **kwargs):
    return super(Test, self).__new__(self, *args, **kwargs)

Instantiating an object from the above class results in this error: TypeError: unbound method __new__() must be called with Test instance as first argument (got ModelBase instance instead).

I can't understand why this is happening (although I know some class magic is happening behind the scenes due to the Django framework).

Any answers will be appreciated.

Answer 1

__new__ doesn't receive an instance as its first parameter. How could it when (a) it's a static method, as you note, and (b) its job is to create an instance and return it! The first parameter of __new__ is conventionally called cls, as it is the class.

Which makes the error message you quote very weird; it is normally the error message you'd get when you called an unbound method (i.e. what you get by accessing ClassName.methodName) with something other than an instance of that class as the self parameter. However, staticmethods (including __new__) don't become unbound methods, they're just simple functions that happen to be attributes of a class:

>>> class Foo(object):
    def __new__(cls, *args, **kwargs):
        return object.__new__(cls)
    def method(self):
        pass

>>> class Bar(object):
    pass

>>> Foo.method
<unbound method Foo.method>

>>> Foo.__new__
<function __new__ at 0x0000000002DB1C88>

>>> Foo.method(Bar())
Traceback (most recent call last):
  File "<pyshell#36>", line 1, in <module>
    Foo.method(Bar())
TypeError: unbound method method() must be called with Foo instance as first argument (got Bar instance instead)

>>> Foo.__new__(Bar)
<__main__.Bar object at 0x0000000002DB4F28>

You can see from this that __new__ should never be an unbound method. Furthermore (unlike a normal method) it doesn't care that you're consistent in what you pass it; I have actually managed to construct an instance of Bar by calling Foo.__new__ because both it and Bar.__new__ are ultimately implemented the same way (deferring all actual work to object.__new__).

However, this led me to look at the source code of Django itself, briefly. Django's Model class has a metaclass, ModelBase. Which is quite complex, and I didn't figure out what it's doing entirely, but I did notice something very interesting.

ModelBase.__new__ (the metaclass __new__, which is the function that creates the class at the end of your class block) invokes its super __new__ without passing it your class dictionary. It instead passes a dictionary with only the __module__ attribute set. Then, after doing a whole bunch of processing, it does the following:

    # Add all attributes to the class.
    for obj_name, obj in attrs.items():
        new_class.add_to_class(obj_name, obj)

(attrs is the dictionary containing all your definitions in your class block, including your __new__ function; add_to_class is a metaclass method that is mostly just setattr).

I'm now 99% certain that the problem is here, because __new__ is a weird implicitly static method. So unlike every other static method, you haven't applied the staticmethod decorator to it. Python (at some level) just recognises the __new__ method and processes it as a static method rather than a normal method[1]. But I'll bet this only happens when you define __new__ in a class block, not when you set it using setattr.

So your __new__, which should be a static method but hasn't been processed by the staticmethod decorator, is being converted into a normal instance method. Then when Python calls it passing the class Test, as per the normal instance creation protocol, it complains that it's not getting an instance of Test.

If that's all correct, then:

1. This fails because Django is a bit broken, but only in that it fails to take Python's inconsistency about __new__ being static into account.
2. You could probably make this work by applying @staticmethod to your __new__ method, even though you shouldn't have to.

---

[1] I believe this is a historical quirk of Python, since __new__ was introduced before the staticmethod decorator, but __new__ can't take an instance, since there's no instance to call it on.