__new__, __init__, and metaclasses (and superclasses)

Question

I've been spending the day trying to understand the intricacies of the python class model, messing around with decorators, metaclasses, and superclasses.

Currently, I'm trying to figure out the role of certain token functions, namely new (back-story here Metaclasses and when/how functions are called)

I've made a new mock-up module to run tests in, here:

#! /usr/bin/env python3

import sys as system
import os  as operating_system

from functools import partial
from time      import perf_counter as counter

class Meta(type):

    @classmethod
    def __prepare__(instance, name, supers, *list, **map):
        print('{} in meta prepare'.format(name))
        return {}

    def __new__(instance, name, supers, attributes, *list, **map):
        print('{} in meta new'.format(name))
        return instance

    def __init__(self, name, supers, attributes, *list, **map):
            print('{} in meta init'.format(self))

    def __call__(self, *list, **map):
        print('{} in meta call'.format(self))
        return type.__call__(self)
        print('after call')

class Super(object):

    def __new__(instance, *list, **map):
        print('{} in Super new'.format(instance))
        return instance

    def __init__(self, *list, **map):
        print('{} in Super init'.format(self))

    def __call__(self, *list, **map):
        print('{} in Super call'.format(self))
        return object.__call__(self)

class Other(object):

    def __new__(instance, *list, **map):
        print('{} in Other new'.format(instance))
        return instance

    def __init__(self, *list, **map):
        print('{} in Other init'.format(self))

    def __call__(self, *list, **map):
        print('{} in Other call'.format(self))
        return object.__call__(self)

class MetaSuper(object, metaclass = Meta):

    def __new__(instance, *list, **map):
        print('{} in MetaSuper new'.format(instance))
        return instance

    def __init__(self, *list, **map):
        print('{} in MetaSuper init'.format(self))

    def __call__(self, *list, **map):
        print('{} in MetaSuper call'.format(self))
        return object.__call__(self)

class DoubleSuper(Super, MetaSuper):

    def __new__(instance, *list, **map):
        print('{} in DoubleSuper new'.format(instance))
        return instance

    def __init__(self, *list, **map):
        print('{} in DoubleSuper init'.format(self))
        Super.__init__(self, *list, **map)
        MetaSuper.__init__(self, *list, **map)

    def __call__(self, *list, **map):
        print('{} in DoubleSuper call'.format(self))
        return object.__call__(self)

class SuperThenMeta(Super, metaclass = Meta):

    def __new__(instance, *list, **map):
        print('{} in SuperThenMeta new'.format(instance))
        return instance

    def __init__(self, *list, **map):
        print('{} in SuperThenMeta init'.format(self))
        Super.__init__(self, *list, **map)

    def __call__(self, *list, **map):
        print('{} in SuperThenMeta call'.format(self))
        return object.__call__(self)

class Triple(Super, Other, metaclass = Meta):

    def __new__(instance, *list, **map):
        print('{} in Triple new'.format(instance))
        return instance

    def __init__(self, *list, **map):
        print('{} in Triple init'.format(self))
        Super.__init__(self, *list, **map)
        Other.__init__(self, *list, **map)

    def __call__(self, *list, **map):
        print('{} in Triple call'.format(self))
        return object.__call__(self)

class Simple(Super):

    def __new__(instance, *list, **map):
        print('{} in Simple new'.format(instance))
        return instance.__init__(instance, *list, **map)

    def __init__(self, *list, **map):
        print('{} in Simple init'.format(self))
        Super.__init__(self, *list, **map)
        Other.__init__(self, *list, **map)

    def __call__(self, *list, **map):
        print('{} in Simple call'.format(self))
        return object.__call__(self)    

def main():
    #thing = SuperThenMeta()
    #other = DoubleSuper()
    last  = Super()
    simp  = Simple()
    trip  = Triple()

if __name__ == '__main__':
    main()

TL;DR, I experimented with a few different setups between these working pieces.

If I run this, this is the output:

MetaSuper in meta prepare
MetaSuper in meta new
SuperThenMeta in meta prepare
SuperThenMeta in meta new
Triple in meta prepare
Triple in meta new
<class '__main__.Super'> in Super new
<class '__main__.Simple'> in Simple new
<class '__main__.Simple'> in Simple init
<class '__main__.Simple'> in Super init
<class '__main__.Simple'> in Other init
Traceback (most recent call last):
File "./metaprogramming.py", line 134, in <module>
  main()
File "./metaprogramming.py", line 131, in main
  trip = Triple()
TypeError: __new__() missing 3 required positional arguments: 'name', 'supers', and 'attributes'

From this, I have a few questions:

• Am I supposed to be calling instance.init(instance, *list, **map) at the end of new functions? I didn't think so, but adding that into the 'Simple' example seemed to work, while 'Super' never reached its init. I was under the impression that by calling object.call in my own call methods, this would be handled by it's default implementation, but no __call__s are made during the whole program.

• Why is calling Triple() calling the metaclasses new first? If this is normal, does that mean this is typical of any class with a metaclass? Is this similar behavior to superclasses?

• I expected call to be in this list somewhere. Does it not get called during the creation routine of objects (eg [prepare], new, init)?

I know this is a lot of information, so thank you for reading this far; any guidance would be appreciated.

Answer 1

Metaclass' __new__

The method __new__ is what is called to create a new instance. Thus its first argument is not an instance, since none has been created yet, but rather the class itself.

In the case of a metaclass, __new__ is expected to return an instance of your metaclass, that is a class. It's signature goes like this:

class Meta(type):
    def __new__(metacls, name, bases, namespace, **kwargs):
        ...

• metacls is the metaclass itself.

• name is a string representing the name of the class being instantiated.

• bases is a tuple of classes from which the class will inherit.

• namespace is the namespace of the class, this is the object returned by __prepare__, now populated with the class attributes.

• **kwargs are any keyword arguments passed to the class at instantiation

To instantiate a class, you need to call type.__new__, which is the default metaclass. You usually do that by calling super().__new__.

class Meta(type):
    def __new__(metacls, name, bases, namespace, **kwargs):
        print('You can do stuff here')

        cls = super().__new__(metacls, name, bases, namespace, **kwargs)

        # You must return the generated class
        return cls

Metaclass' __init__

The __init__ method behaves the same as for any other class. It receives the created instance, here a class, as argument if __new__ returned an instance of the expected type. In your example, __new__ does not return an object of type Meta. It return Meta itself which is of type type.

The following __init__ method is never called on instantiation.

class Meta(type):
    def __new__(metacls, name, bases, namespace, **kwargs):
        return None # or anything such that type(obj) is not Meta
    
    def __init__(self, name, bases, namespace, **kwargs):
        # This will never be called because the return type of `__new__` is wrong
        pass
        

The following is called on instantiation, because Meta.__new__ correctly returns an object of type Meta.

class Meta(type):
        def __new__(metacls, name, bases, namespace, **kwargs):
            return super().__new__(metacls, name, bases, namespace, **kwargs)
        
        def __init__(self, name, bases, namespace, **kwargs):
            print('__init__ was called')

Metaclass' __call__

Again, the behaviour of __call__ is no different than for any other class. It is called when you try to call an instance of the metaclass, while __new__ and __init__ are called when you called the metaclass to create an instance (a class).

Of course, calling a class is expected to return an instance, so do not forget to call super().__call__ and return its result, otherwise you would be short-circuiting the instance creation, as it is type.__call__ which calls __new__ and __init__.

class Meta(type):
    def __call__(self, *args, **kwargs):
        print(f'An instance was called with {args}')
        return super().__call__(self, *args, **kwargs)

# This declaration if what calls __new__ and __init__ of the metaclass
class Klass(metaclass=Meta):
    pass

# This calls the __call__ method of the metaclass
instance = Klass()