Arguments of __new__ and __init__ for metaclasses

Question

I am a bit surprised by the method call order and the different arguments when overriding new and init in a metaclass. Consider the following:

class AT(type):
    def __new__(mcs, name, bases, dct):
        print(f"Name as received in new: {name}")
        return super().__new__(mcs, name + 'HELLO', bases + (list,), dct)

    def __init__(cls, name, bases, dct):
        print(f"Name as received in init: {name}")
        pass

class A(metaclass=AT):
    pass

A.__name__

The output is:

Name as received in new: A
Name as received in init: A
'AHELLO'

In short I would have expected init to receive AHELLO with the argument name.

I imagined that __init__ was called by super().__new__: if the call is not done in the overridden __new__ then my __init__ is not called.

Could someone clarify how __init__ is called in this case?

For information my use case for this is that I wanted to make creation of classes, in a special case, easier at runtime by providing only a single "base" class (and not a tuple), I then added this code in __new__:

if not isinstance(bases, tuple):
            bases = (bases, )

however, I found out that I also need to add it in __init__.

Answer 1

Your __init__ method is obviously called and the reason for that is because your __new__ method is returning an instance of your class.

From https://docs.python.org/3/reference/datamodel.html#object.new:

If __new__() returns an instance of cls, then the new instance’s __init__() method will be invoked like __init__(self[, ...]), where self is the new instance and the remaining arguments are the same as were passed to __new__().

As you can see the arguments passed to __init__ are those passed to __new__ method's caller not when you call it using super. It's a little bit vague but that's what it means if you read it closely.

And regarding the rest it works just as expected:

In [10]: A.__bases__
Out[10]: (list,)

In [11]: a = A()

In [12]: a.__class__.__bases__
Out[12]: (list,)

Answer 2

The fact is that what orchestrates the call of __new__ and __init__ of an ordinary class is the __call__ method on its metaclass. The code in the __call__ method of type, the default metatype, is in C, but the equivalent of it in Python would be:

class type:
    ...
    def __call__(cls, *args, **kw):
         instance = cls.__new__(cls, *args, **kw)  # __new__ is actually a static method - cls has to be passed explicitly
         if isinstance(instance, cls):
               instance.__init__(*args, **kw)
         return instance

That takes place for most object instantiation in Python, including when instantiating classes themselves - the metaclass is implicitly called as part of a class statement. In this case, the __new__ and __init__ called from type.__call__ are the methods on the metaclass itself. And in this case, type is acting as the "metametaclass" - a concept seldom needed, but it is what creates the behavior you are exploring.

When creating classes, type.__new__ will be responsible for calling the class (not the metaclass) __init_subclass__, and its descriptors' __set_name__ methods - so, the "metametaclass" __call__ method can't control that.

So, if you want the args passed to the metaclass __init__ to be programmatically modified, the "normal" way will be to have a "metametaclass", inheriting from type and distinct from your metaclass itself, and override its __call__ method:

class MM(type):
    def __call__(metacls, name, bases, namespace, **kw):
        name = modify(name)
        cls = metacls.__new__(metacls, name, bases, namespace, **kw)
        metacls.__init__(cls, name, bases, namespace, **kw)
        return cls
        # or you could delegate to type.__call__, replacing the above with just
        # return super().__call__(modify(name), bases, namespace, **kw)

Of course that is a way to get to go closer to "turtles all way to the bottom" than anyone would ever like in production code.

An alternative is to keep the modified name as an attribute on the metaclass, so that its __init__ method can take the needed information from there, and ignore the name passed in from its own metaclass' __call__ invocation. The information channel can be an ordinary attribute on the metaclass instance. Well - it happens that the "metaclass instance" is the class being created itself - and oh, see - that the name passed to type.__new__ already gets recorded in it - on the __name__ atribute.

In other words, all you have to do to use a class name modified in a metaclass __new__ method in its own __init__ method, is to ignore the passed in name argument, and use cls.__name__ instead:

class Meta(type):
    def __new__(mcls, name, bases, namespace, **kw):
        name = modified(name)
        return super().__new__(mcls, name, bases, namespace, **kw)

    def __init__(cls, name, bases, namespace, **kw):
        name = cls.__name__  # noQA  (otherwise linting tools would warn on the overriden parameter name)
        ...