The call order of python3 metaclass

Question

I am in confusion when trying to understand the order that metaclass creates a class instance. According to this diagram (source),

I type the following codes to verify it.

class Meta(type):
    def __call__(self):
        print("Meta __call__")
        super(Meta, self).__call__()

    def __new__(mcs, name, bases, attrs, **kwargs):
        print("Meta __new__")
        return super().__new__(mcs, name, bases, kwargs)

    def __prepare__(msc, name, **kwargs):
        print("Meta __prepare__")
        return {}

class SubMeta(Meta):
    def __call__(self):
        print("SubMeta __call__!")
        super().__call__()

    def __new__(mcs, name, bases, attrs, **kwargs):
        print("SubMeta __new__")
        return super().__new__(mcs, name, bases, kwargs)

    def __prepare__(msc, name, **kwargs):
        print("SubMeta __prepare__")
        return Meta.__prepare__(name, kwargs)

class B(metaclass = SubMeta):
    pass

b = B()

However, the result seems not like this follow.

SubMeta __prepare__
Meta __prepare__
SubMeta __new__
Meta __new__
SubMeta __call__!
Meta __call__

Any help will be appreciated.

Answer 1

The trick, identified

Update 2: Based on behavior, the fact that M0.__call__ is called below must be a side effect of this line in builtin__build_class in the CPython source (Python/bltinmodule.c).

In order to define a class that has a metaclass, we call the metaclass's __prepare__, __new__, and __init__ as usual. This creates a class—in the example below, Meta—that is callable, but its internal PyFunction_GET_CODE slot points not to its own __call__ but rather to its metaclass's __call__. Hence if we call Meta() (the metaclass object), we invoke M0.__call__:

print("call Meta")
print("Meta returns:", Meta('name', (), {}))
print("finished calling Meta")

produces:

call Meta
M0 __call__: mmcls=<class '__main__.Meta'>, args=('name', (), {}), kwargs={}
Meta __new__: mcs=<class '__main__.Meta'>, name='name', bases=(), attrs={}, kwargs={}
Meta __init__: mcs=<class '__main__.name'>, name='name', bases=(), attrs={}, kwargs={}
Meta returns: <class '__main__.name'>
finished calling Meta

In other words, we see that Meta acts like type, but it (rather magically and not very well documented) invokes M0.__call__. This is no doubt due to looking up __call__ in the class's type, rather than in an instance of the class (and indeed there is no instance except for the one we're creating). This is in fact the general case: it falls out of the fact that we call __call__ on the type of Meta, and the type of Meta is M0:

print("type(Meta) =", type(Meta))

prints:

type(Meta) = <class '__main__.M0'>

which explains where this comes from. (I still think this should be emphasized in the documentation, which also should describe the constraints on metaclass typing—these are enforced in _calculate_winner in Lib/types.py and, as C code, in _PyType_CalculateMetaclass in Objects/typeobject.c.)

Updated original answer

I don't know where your diagram came from, but it's wrong. UPDATE: You can in fact have a metaclass for your metaclass; see jsbueno's answer, and I've updated the example below. New sentences / text are in bold, except for the final section describing my puzzlement at the apparent lack of documentation.

Your existing metaclass code has at least one error. Most significantly, its __prepare__ needs to be a class-method. See also Using the __call__ method of a metaclass instead of __new__? and PEP 3115. And, to use a meta-meta-class, your metaclass needs to have a metaclass of its own, not a base class.

Chris's answer contains correct definitions. But there are some unfortunate asymmetries between metaclass method arguments and class method arguments, which I'll illustrate below.

One other thing that may help: note that the metaclass __prepare__ method is called before creating any instances of class B: it is called when class B itself is being defined. To show this, here is a corrected metaclass-and-class. I have also added a few more illustrators. I've added a meta-metaclass as well, based on jsbueno's answer. I cannot find formal Python documentation on this, but I've updated the output below.

class M0(type):
    def __call__(mmcls, *args, **kwargs):
        print("M0 __call__: mmcls={!r}, "
              "args={!r}, kwargs={!r}".format(mmcls, args, kwargs))
        return super().__call__(*args, **kwargs)

class Meta(type, metaclass=M0):
    def __call__(cls, *args, **kwargs):
        print("Meta __call__: cls={!r}, "
              "args={!r}, kwargs={!r}".format(cls, args, kwargs))
        return super().__call__(*args, **kwargs)

    def __new__(mcs, name, bases, attrs, **kwargs):
        print("Meta __new__: mcs={!r}, name={!r}, bases={!r}, "
              "attrs={!r}, kwargs={!r}".format(mcs, name, bases, attrs, kwargs))
        return super().__new__(mcs, name, bases, attrs)

    def __init__(mcs, name, bases, attrs, **kwargs):
        print("Meta __init__: mcs={!r}, name={!r}, bases={!r}, "
              "attrs={!r}, kwargs={!r}".format(mcs, name, bases, attrs, kwargs))
        super().__init__(name, bases, attrs, **kwargs)

    @classmethod
    def __prepare__(cls, name, bases, **kwargs):
        print("Meta __prepare__: name={!r}, "
              "bases={!r}, kwargs={!r}".format(name, bases, kwargs))
        return {}

print("about to create class A")
class A(metaclass=Meta): pass
print("finished creating class A")

print("about to create class B")

class B(A, metaclass=Meta, foo=3):
    @staticmethod
    def __new__(cls, *args, **kwargs):
        print("B __new__: cls={!r}, "
              "args={!r}, kwargs={!r}".format(cls, args, kwargs))
        return super().__new__(cls)

    def __init__(self, *args, **kwargs):
        print("B __init__: args={!r}, kwargs={!r}, ".format(args, kwargs))

print("finished creating class B")

print("about to create instance b = B()")
b = B('hello', bar=7)
print("finished creating instance b")

Now, let's observe what happens when I run this, and take each piece apart:

$ python3.6 meta.py
about to create class A
Meta __prepare__: name='A', bases=(), kwargs={}
M0 __call__: mmcls=<class '__main__.Meta'>, args=('A', (), {'__module__': '__main__', '__qualname__': 'A'}), kwargs={}
Meta __new__: mcs=<class '__main__.Meta'>, name='A', bases=(), attrs={'__module__': '__main__', '__qualname__': 'A'}, kwargs={}
Meta __init__: mcs=<class '__main__.A'>, name='A', bases=(), attrs={'__module__': '__main__', '__qualname__': 'A'}, kwargs={}
finished creating class A

To create class A itself, Python first calls the metaclass's __prepare__, delivering it the name of the class (A), the list of base classes (an empty tuple—it's called a list but is actually a tuple), and any keyword arguments (none). As PEP 3115 notes, the metaclass needs to return a dictionary or dict-like object; this one does by just returning an empty dictionary, so we are good here.

(I don't print cls itself here, but if you do, you will see it is just <class '__main__.Meta'>.)

Next, having gotten a dictionary from __prepare__, Python first calls the meta-meta __call__, i.e., M0.__call__, passing the entire set of arguments as the args tuple. It then populates the __prepare__-supplied dictionary with all the attributes for the class, passing this as the attrs to the metaclass __new__ and __init__. If you print the id of the dictionary returned from __prepare__ and passed to __new__ and __init__ you will see they all match.

Since class A has no methods or data members, we see only the magic __module__ and __qualname__ attributes here. We also see no keyword arguments, so now let's move on to creating class B:

about to create class B
Meta __prepare__: name='B', bases=(<class '__main__.A'>,), kwargs={'foo': 3}
M0 __call__: mmcls=<class '__main__.Meta'>, args=('B', (<class '__main__.A'>,), {'__module__': '__main__', '__qualname__': 'B', '__new__': <staticmethod object at 0x800ad0a58>, '__init__': <function B.__init__ at 0x800ad2840>, '__classcell__': <cell at 0x800a749d8: empty>}), kwargs={'foo': 3}
Meta __new__: mcs=<class '__main__.Meta'>, name='B', bases=(<class '__main__.A'>,), attrs={'__module__': '__main__', '__qualname__': 'B', '__new__': <staticmethod object at 0x800ad0940>, '__init__': <function B.__init__ at 0x800ad27b8>, '__classcell__': <cell at 0x800a745b8: empty>}, kwargs={'foo': 3}
Meta __init__: mcs=<class '__main__.B'>, name='B', bases=(<class '__main__.A'>,), attrs={'__module__': '__main__', '__qualname__': 'B', '__new__': <staticmethod object at 0x800ad0940>, '__init__': <function B.__init__ at 0x800ad27b8>, '__classcell__': <cell at 0x800a745b8: Meta object at 0x802047018>}, kwargs={'foo': 3}
finished creating class B

This one is rather more interesting. Now we have one base class, namely __main__.A. Class B also defines several methods (__new__ and __init__) and we see them in the attrs dictionaries passed to the metaclass __new__ and __init__ methods (which, remember, are just the now-populated dictionary returned by the metaclass's __prepare__). As before, the passing-on happens through the meta-meta-class M0.__call__. We also see one keyword argument throughout, {'foo': 3}. In the attribute dictionary, we can also observe the magic __classcell__ entry: see Provide __classcell__ example for Python 3.6 metaclass for a short description as to what this is about, but to be, er, super-short, it's for making super() work.

The keyword argument is passed to all three metaclass methods, plus that of the meta-meta-class. (I'm not quite sure why. Note that modifying the dictionary in any metaclass method does not affect it in any other, as it's a copy each time of the original keyword arguments. However, we can modify it in the meta-meta-class: add kwargs.pop('foo', None) to M0.__call__ to observe this.)

Now that we have our classes A and B, we can get on to the process of creating an actual instance of class B. Now we see the metaclass's __call__ invoked (not the meta-meta-class's):

about to create instance b = B()
Meta __call__: cls=<class '__main__.B'>, args=('hello',), kwargs={'bar': 7}

It's possible to change the args or kwargs passed on, but I don't; the sample code above winds up calling type.__call__(cls, *args, **kwargs) (through the magic of super().__call__). This in turn calls B.__new__ and B.__init__:

B __new__: cls=<class '__main__.B'>, args=('hello',), kwargs={'bar': 7}
B __init__: args=('hello',), kwargs={'bar': 7}, 
finished creating instance b

which finishes the realization of the new instance of class B, which we then bind to the name b.

Note that B.__new__ says:

return super().__new__(cls)

so we invoke object.__new__ to create the instance—this is more or less a requirement of all versions of Python; you can only "cheat" when you return a singleton instance (ideally, one that's non-modifiable). It's type.__call__ that calls B.__init__ on this object, passing the arguments and keyword-arguments we passed it. If we replace Meta's __call__ with:

    def __call__(cls, *args, **kwargs):
        print("Meta __call__: cls={!r}, "
              "args={!r}, kwargs={!r}".format(cls, args, kwargs))
        return object.__new__(cls)

we will see that B.__new__ and B.__init__ are never called:

about to create instance b = B()
Meta __call__: cls=<class '__main__.B'>, args=('hello',), kwargs={'bar': 7}
finished creating instance b

This would, in effect, create a useless/uninitialized instance b. It's therefore critical that the metaclass __call__ method call the underlying class's __init__, usually by invoking type.__call__ via super().__call__. If the underlying class has a __new__, the metaclass should call that first, again usually by invoking type.__call__.

Side note: what the documentation says

To quote section 3.3.3.6:

Once the class namespace has been populated by executing the class body, the class object is created by calling metaclass(name, bases, namespace, **kwds) (the additional keywords passed here are the same as those passed to __prepare__).

This explains the call to Meta.__call__ when creating b as an instance of class B, but not the fact that Python first calls M0.__call__ before calling Meta.__new__ and Meta.__init__ when creating classes A and B themselves.

The next paragraph mentions the __classcell__ entry; the one after that goes on to describe the use of __set_name__ and __init_subclass__ hooks. Nothing here tells us how or why Python calls M0.__call__ at this point.

Earlier, in sections 3.3.3.3 through 3.3.3.5, the documentation describes the process of determining the metaclass, preparing the class namespace, and executing the class body. This is where the meta-metaclass action should be described, but isn't.

Several additional sections describe a few additional constraints. One important one is 3.3.10, which talks about how special methods are found via the object type, bypassing both regular member attribute lookups and even (sometimes) a metaclass getattribute, saying:

Bypassing the __getattribute__() machinery in this fashion provides significant scope for speed optimisations within the interpreter, at the cost of some flexibility in the handling of special methods (the special method must be set on the class object itself in order to be consistently invoked by the interpreter).

Update 2: This is really the secret of the trick: the special __call__ method is found via the type's type. If the metaclass has a metaclass, the meta-meta-class provides the __call__ slot; otherwise the type of the metaclass is type, so that the __call__ slot is type.__call__.

Answer 2

Despite @torek's lenghty answer, with a lot of other details on class creation, what you brought together to this question is mostly correct.

The only thing that is wrong in your code, which propably puzzled you is that te class you call Meta have to be itself the metaclass from SubMeta and not its parent.

Simply change Submeta declaration to:

class SubMeta(type, metaclass=Meta):
    ...

(No need for it to inherit from "Meta" as well - it can derive only from type. It is otherwise though to think of a customization to type.__call__ that would be usefull at the same time for creating instances of your classes (that is when SubMeta.__call__ is called), and your classes themselves (Meta.__call__ called))

Here is another, shorter example I just typed at the terminal. Sorry for the naming inconsistencies, and for being less complete - but it shows the main point:

class M(type):
    def __call__(mmcls, *args, **kwargs):
        print("M's call", args, kwargs)
        return super().__call__(*args, **kwargs)

class MM(type, metaclass=M):
    def __prepare__(cls, *args, **kw):
        print("MM Prepare")
        return {}
    def __new__(mcls, *args, **kw):
        print("MM __new__")
        return super().__new__(mcls, *args, **kw)

class klass(metaclass=MM):
    pass

Upon processing the klass body, Python output was:

MM Prepare
M's call ('klass', (), {'__module__': '__main__', '__qualname__': 'klass'}) {}
MM __new__

Moreover

As you can see from this, with a meta-meta class it is possible to customize the call order and parameters to the metaclass __init__ and __new__, but there are still steps that can't be customized from pure-Python code, and would require native calls to API's (and possibly raw object structure manipulation) - that are:

• One can't control the call to __prepare__
• One can't control the call to __init_subclass__ on the created classes
• One can control when descriptors' __set_name__ are called

The last two items take place after meta-meta's __call__ return, and before resuming the flow to the module where the class module is.