Behavior of __new__ in a metaclass (also in context of inheritance)

Question

Ok, obviously __new__ in a metaclass runs when an instance of the metaclass i.e. a class object is instantiated, so __new__ in a metaclass provides a hook to intercept events (/code that runs) at class definition time (e.g. validating/enforcing rules for class attributes such as methods etc.).

Many online examples of __new__ in a metaclass return an instance of the type constructor from __new__, which seems a bit problematic since this blocks __init__ (docs: "If __new__() does not return an instance of cls, then the new instance’s __init__() method will not be invoked").

While tinkering with return values of __new__ in a metaclass I came across some somewhat strange cases which I do not fully understand, e.g.:

class Meta(type):
    
    def __new__(self, name, bases, attrs):
        print("Meta __new__ running!")
        # return type(name, bases, attrs)                     # 1. 
        # return super().__new__(self, name, bases, attrs)    # 2.
        # return super().__new__(name, bases, attrs)          # 3.  
        # return super().__new__(type, name, bases, attrs)    # 4. 
        # return self(name, bases, attrs)                     # 5.

    def __init__(self, *args, **kwargs):
        print("Meta __init__ running!")
        return super().__init__(*args, **kwargs)
    
class Cls(metaclass=Meta):
    pass

1. This is often seen in examples and generally works, but blocks __init__
2. This works and __init__ also fires; but why pass self to a super() call? Shouldn't self/cls get passed automatically with super()?
3. This throws a somewhat strange error I can't really make sense of: TypeError: type.__new__(X): X is not a type object (str); what is X? shouldn't self be auto-passed?
4. The error of 3. inspired me to play with the first arg of the super() call, so I tried to pass type directly; this also blocks __init__. What is happening here?
5. tried just for fun; this leads to a RecursionError

Also especially cases 1. and 2. appear to have quite profound implications for inheriting from classes bound to metaclasses:

class LowerCaseEnforcer(type):
    """ Allows only lower case names as class attributes! """

    def __new__(self, name, bases, attrs): 
        for name in attrs:
            if name.lower() != name:
                raise TypeError(f"Inappropriate method name: {name}")
            
        # return type(name, bases, attrs)                    # 1.
        # return super().__new__(self, name, bases, attrs)   # 2.

    class Super(metaclass=LowerCaseEnforcer):
        pass
    
    class Sub(Super):
        
        def some_method(self):
            pass
    
        ## this will error in case 2 but not case 1
        def Another_method(self):
            pass

1. expected behavior: metaclass is bound to superclass, but not to subclass
2. binds the superclass /and/ subclasses to the metaclass; ?

I would much appreciate if someone could slowly and kindly explain what exactly is going on in the above examples! :D

Answer 1

It is simpler than what you got too.

As you have noted, the correct thing to do is your 2 above:

return super().__new__(self, name, bases, attrs)    # 2.

Here it goes: __new__ is a special method - although in certain documentations, even part of the official documentation, it is described as being a classmethod, it is not quite so: it, as an object, behaves more like a static method - in a sense that Python does not automatically fill the first parameter when one calls MyClass.__new__() - i.e., you'd have to call MyClass.__new__(MyClass) for it to work. (I am a step back here - this info applies to all classes: metaclasses and ordinary classes).

When you call MyClass() to create a new instance, then Python will call MyClass.__new__ and insert the cls parameter as first parameter.

With metaclasses, the call to create a new instance of the metaclass is triggered by the execution of the class statement and its class body. Likewise, Python fills in the first parameter to Metaclass.__new__, passing the metaclass itself.

When you call super().__new__ from within your metaclass' __new__ you are in the same case of one calling __new__ manually: the parameter specifying which class' that __new__ should apply have to be explicitly filled.

Now, what is confusing you is that you are writting the first parameter to __new__ as self - which would be correct if it were an instance of the metaclass (i.e. an ordinary class). As it is, that parameter is a reference to the metaclass itself.

The docs does not inform an official, or recomended name for the first parameter of a metaclass __new__, but usually it is something along mcls, mcs, metaclass, metacls - to make it different from cls which is the usuall name for the first parameter of a non-metaclass __new__ method. In a metaclass, the "class" - cls is what is created by the ultimate call to type.__new__ (either hardcoded, or using super()) the return of it is the new-born class (it can be further modified in the __new__ method after the call to the superclass) - and when returned, the call to __init__ takes place normally.

So, I will just comment further the use of trying to call type(name, bases, namespace) instead of type.__new__(mcls, name, bases, namespace): the first form will just create a plain class, as if the metaclass had not been used at all - (lines in the metaclass __new__ that modify the namespace or bases, of course, have their effect. But the resulting class will have type as its metaclass, and subclasses of it won't call the metaclass at all. (For the record, it works as a "pre-class decorator" - which can act on class parameters before it is created, and it could even be an ordinary function, instead of a class with a __new__ method - the call to type is what will create the new class after all)

A simple way to check if the metaclass is "bound" to your class is to check its type with type(MyClass) or MyClass.__class__ .