Why do keyword arguments to a class definition reappear after they were removed?

Question

I created a metaclass that defines the __prepare__ method, which is supposed to consume a specific keyword in the class definition, like this:

class M(type):
    @classmethod
    def __prepare__(metaclass, name, bases, **kwds):
        print('in M.__prepare__:')
        print(f'  {metaclass=}\n  {name=}\n'
              f'  {bases=}\n  {kwds=}\n  {id(kwds)=}')
        if 'for_prepare' not in kwds:
            return super().__prepare__(name, bases, **kwds)
        arg = kwds.pop('for_prepare')
        print(f'  arg popped for prepare: {arg}')
        print(f'  end of prepare: {kwds=} {id(kwds)=}')
        return super().__prepare__(name, bases, **kwds)

    def __new__(metaclass, name, bases, ns, **kwds):
        print('in M.__new__:')
        print(f'  {metaclass=}\n  {name=}\n'
              f'  {bases=}\n  {ns=}\n  {kwds=}\n  {id(kwds)=}')
        return super().__new__(metaclass, name, bases, ns, **kwds)


class A(metaclass=M, for_prepare='xyz'):
    pass

When I run it, the for_prepare keyword argument in the definition of class A reappears in __new__ (and later in __init_subclass__, where it causes an error):

$ python3 ./weird_prepare.py
in M.__prepare__:
  metaclass=<class '__main__.M'>
  name='A'
  bases=()
  kwds={'for_prepare': 'xyz'}
  id(kwds)=140128409916224
  arg popped for prepare: xyz
  end of prepare: kwds={} id(kwds)=140128409916224
in M.__new__:
  metaclass=<class '__main__.M'>
  name='A'
  bases=()
  ns={'__module__': '__main__', '__qualname__': 'A'}
  kwds={'for_prepare': 'xyz'}
  id(kwds)=140128409916224
Traceback (most recent call last):
  File "./weird_prepare.py", line 21, in <module>
    class A(metaclass=M, for_prepare='xyz'):
  File "./weird_prepare.py", line 18, in __new__
    return super().__new__(metaclass, name, bases, ns, **kwds)
TypeError: __init_subclass__() takes no keyword arguments

As you can see the for_prepare item is removed from the dict, and the dict that is passed to __new__ is the same object that was passed to __prepare__ and the same object that the for_prepare item was popped from, but in __new__ it reappeared! Why does a keyword that was deleted from the dict get added back in?

Answer 1

and the dict that is passed to new is the same object that was passed to prepare

Unfortunately, this is where you are wrong.

Python only recycles the same object id.

If you create a new dict inside __prepare__ you will notice the id of kwds changes in __new__.

class M(type):
    @classmethod
    def __prepare__(metaclass, name, bases, **kwds):
        print('in M.__prepare__:')
        print(f'  {metaclass=}\n  {name=}\n'
              f'  {bases=}\n  {kwds=}\n  {id(kwds)=}')
        if 'for_prepare' not in kwds:
            return super().__prepare__(name, bases, **kwds)
        arg = kwds.pop('for_prepare')
        x = {} # <<< create a new dict
        print(f'  arg popped for prepare: {arg}')
        print(f'  end of prepare: {kwds=} {id(kwds)=}')
        return super().__prepare__(name, bases, **kwds)

    def __new__(metaclass, name, bases, ns, **kwds):
        print('in M.__new__:')
        print(f'  {metaclass=}\n  {name=}\n'
              f'  {bases=}\n  {ns=}\n  {kwds=}\n  {id(kwds)=}')
        return super().__new__(metaclass, name, bases, ns, **kwds)


class A(metaclass=M, for_prepare='xyz'):
    pass

Output:

in M.__prepare__:
  metaclass=<class '__main__.M'>
  name='A'
  bases=()
  kwds={'for_prepare': 'xyz'}
  id(kwds)=2595838763072
  arg popped for prepare: xyz
  end of prepare: kwds={} id(kwds)=2595838763072
in M.__new__:
  metaclass=<class '__main__.M'>
  name='A'
  bases=()
  ns={'__module__': '__main__', '__qualname__': 'A'}
  kwds={'for_prepare': 'xyz'}
  id(kwds)=2595836298496 # <<< id has changed now
Traceback (most recent call last):
  File "d:\nemetris\mpf\mpf.test\test_so4.py", line 22, in <module>
    class A(metaclass=M, for_prepare='xyz'):
  File "d:\nemetris\mpf\mpf.test\test_so4.py", line 19, in __new__ 
    return super().__new__(metaclass, name, bases, ns, **kwds)     
TypeError: A.__init_subclass__() takes no keyword arguments        

Answer 2

This is not an effect of metaclasses, but of **kwargs. Whenever a function is called with **kwargs, the current dict is unpacked and not passed on. Whenever a function receives **kwargs, a new dict is created.

In effect, when both of caller/callee use **kwargs then the dict seen by either is a copy.

Compare the setup of using **kwargs in isolation:

def first(**kwargs):
    print(f"Popped 'some_arg': {kwargs.pop('some_arg')!r}")

def second(**kwargs):
    print(f"Got {kwargs} in the end")

def head(**kwargs):
    first(**kwargs)
    second(**kwargs)

head(a=2, b=3, some_arg="Watch this!", c=4)
# Popped 'some_arg': 'Watch this!'
# Got {'a': 2, 'b': 3, 'some_arg': 'Watch this!', 'c': 4} in the end

Likewise, __prepare__ and __new__ are separately called when creating a class. Their **kwargs are shallow copies and neither adding nor removing items is visible to the other call.