What are some (concrete) use-cases for metaclasses?

Question

I was asked the same question recently, and came up with several answers. I hope it's OK to revive this thread, as I wanted to elaborate on a few of the use cases mentioned, and add a few new ones.

Most metaclasses I've seen do one of two things:

1. Registration (adding a class to a data structure):

   models = {}
   
   class ModelMetaclass(type):
       def __new__(meta, name, bases, attrs):
           models[name] = cls = type.__new__(meta, name, bases, attrs)
           return cls
   
   class Model(object):
       __metaclass__ = ModelMetaclass
   

   Whenever you subclass Model, your class is registered in the models dictionary:

   >>> class A(Model):
   ...     pass
   ...
   >>> class B(A):
   ...     pass
   ...
   >>> models
   {'A': <__main__.A class at 0x...>,
    'B': <__main__.B class at 0x...>}
   

   This can also be done with class decorators:

   models = {}
   
   def model(cls):
       models[cls.__name__] = cls
       return cls
   
   @model
   class A(object):
       pass
   

   Or with an explicit registration function:

   models = {}
   
   def register_model(cls):
       models[cls.__name__] = cls
   
   class A(object):
       pass
   
   register_model(A)
   

   Actually, this is pretty much the same: you mention class decorators unfavorably, but it's really nothing more than syntactic sugar for a function invocation on a class, so there's no magic about it.

   Anyway, the advantage of metaclasses in this case is inheritance, as they work for any subclasses, whereas the other solutions only work for subclasses explicitly decorated or registered.

   >>> class B(A):
   ...     pass
   ...
   >>> models
   {'A': <__main__.A class at 0x...> # No B :(
   

2. Refactoring (modifying class attributes or adding new ones):

   class ModelMetaclass(type):
       def __new__(meta, name, bases, attrs):
           fields = {}
           for key, value in attrs.items():
               if isinstance(value, Field):
                   value.name = '%s.%s' % (name, key)
                   fields[key] = value
           for base in bases:
               if hasattr(base, '_fields'):
                   fields.update(base._fields)
           attrs['_fields'] = fields
           return type.__new__(meta, name, bases, attrs)
   
   class Model(object):
       __metaclass__ = ModelMetaclass
   

   Whenever you subclass Model and define some Field attributes, they are injected with their names (for more informative error messages, for example), and grouped into a _fields dictionary (for easy iteration, without having to look through all the class attributes and all its base classes' attributes every time):

   >>> class A(Model):
   ...     foo = Integer()
   ...
   >>> class B(A):
   ...     bar = String()
   ...
   >>> B._fields
   {'foo': Integer('A.foo'), 'bar': String('B.bar')}
   

   Again, this can be done (without inheritance) with a class decorator:

   def model(cls):
       fields = {}
       for key, value in vars(cls).items():
           if isinstance(value, Field):
               value.name = '%s.%s' % (cls.__name__, key)
               fields[key] = value
       for base in cls.__bases__:
           if hasattr(base, '_fields'):
               fields.update(base._fields)
       cls._fields = fields
       return cls
   
   @model
   class A(object):
       foo = Integer()
   
   class B(A):
       bar = String()
   
   # B.bar has no name :(
   # B._fields is {'foo': Integer('A.foo')} :(
   

   Or explicitly:

   class A(object):
       foo = Integer('A.foo')
       _fields = {'foo': foo} # Don't forget all the base classes' fields, too!
   

   Although, on the contrary to your advocacy for readable and maintainable non-meta programming, this is much more cumbersome, redundant and error prone:

   class B(A):
       bar = String()
   
   # vs.
   
   class B(A):
       bar = String('bar')
       _fields = {'B.bar': bar, 'A.foo': A.foo}
   

Having considered the most common and concrete use cases, the only cases where you absolutely HAVE to use metaclasses are when you want to modify the class name or list of base classes, because once defined, these parameters are baked into the class, and no decorator or function can unbake them.

class Metaclass(type):
    def __new__(meta, name, bases, attrs):
        return type.__new__(meta, 'foo', (int,), attrs)

class Baseclass(object):
    __metaclass__ = Metaclass

class A(Baseclass):
    pass

class B(A):
    pass

print A.__name__ # foo
print B.__name__ # foo
print issubclass(B, A)   # False
print issubclass(B, int) # True

This may be useful in frameworks for issuing warnings whenever classes with similar names or incomplete inheritance trees are defined, but I can't think of a reason beside trolling to actually change these values. Maybe David Beazley can.

Anyway, in Python 3, metaclasses also have the __prepare__ method, which lets you evaluate the class body into a mapping other than a dict, thus supporting ordered attributes, overloaded attributes, and other wicked cool stuff:

import collections

class Metaclass(type):

    @classmethod
    def __prepare__(meta, name, bases, **kwds):
        return collections.OrderedDict()

    def __new__(meta, name, bases, attrs, **kwds):
        print(list(attrs))
        # Do more stuff...

class A(metaclass=Metaclass):
    x = 1
    y = 2

# prints ['x', 'y'] rather than ['y', 'x']

 

class ListDict(dict):
    def __setitem__(self, key, value):
        self.setdefault(key, []).append(value)

class Metaclass(type):

    @classmethod
    def __prepare__(meta, name, bases, **kwds):
        return ListDict()

    def __new__(meta, name, bases, attrs, **kwds):
        print(attrs['foo'])
        # Do more stuff...

class A(metaclass=Metaclass):

    def foo(self):
        pass

    def foo(self, x):
        pass

# prints [<function foo at 0x...>, <function foo at 0x...>] rather than <function foo at 0x...>

You might argue ordered attributes can be achieved with creation counters, and overloading can be simulated with default arguments:

import itertools

class Attribute(object):
    _counter = itertools.count()
    def __init__(self):
        self._count = Attribute._counter.next()

class A(object):
    x = Attribute()
    y = Attribute()

A._order = sorted([(k, v) for k, v in vars(A).items() if isinstance(v, Attribute)],
                  key = lambda (k, v): v._count)

 

class A(object):

    def _foo0(self):
        pass

    def _foo1(self, x):
        pass

    def foo(self, x=None):
        if x is None:
            return self._foo0()
        else:
            return self._foo1(x)

Besides being much more ugly, it's also less flexible: what if you want ordered literal attributes, like integers and strings? What if None is a valid value for x?

Here's a creative way to solve the first problem:

import sys

class Builder(object):
    def __call__(self, cls):
        cls._order = self.frame.f_code.co_names
        return cls

def ordered():
    builder = Builder()
    def trace(frame, event, arg):
        builder.frame = frame
        sys.settrace(None)
    sys.settrace(trace)
    return builder

@ordered()
class A(object):
    x = 1
    y = 'foo'

print A._order # ['x', 'y']

And here's a creative way to solve the second one:

_undefined = object()

class A(object):

    def _foo0(self):
        pass

    def _foo1(self, x):
        pass

    def foo(self, x=_undefined):
        if x is _undefined:
            return self._foo0()
        else:
            return self._foo1(x)

But this is much, MUCH voodoo-er than a simple metaclass (especially the first one, which really melts your brain). My point is, you look at metaclasses as unfamiliar and counter-intuitive, but you can also look at them as the next step of evolution in programming languages: you just have to adjust your mindset. After all, you could probably do everything in C, including defining a struct with function pointers and passing it as the first argument to its functions. A person seeing C++ for the first time might say, "what is this magic? Why is the compiler implicitly passing this to methods, but not to regular and static functions? It's better to be explicit and verbose about your arguments". But then, object-oriented programming is much more powerful once you get it; and so is this, uh... quasi-aspect-oriented programming, I guess. And once you understand metaclasses, they're actually very simple, so why not use them when convenient?

And finally, metaclasses are rad, and programming should be fun. Using standard programming constructs and design patterns all the time is boring and uninspiring, and hinders your imagination. Live a little! Here's a metametaclass, just for you.

class MetaMetaclass(type):
    def __new__(meta, name, bases, attrs):
        def __new__(meta, name, bases, attrs):
            cls = type.__new__(meta, name, bases, attrs)
            cls._label = 'Made in %s' % meta.__name__
            return cls 
        attrs['__new__'] = __new__
        return type.__new__(meta, name, bases, attrs)

class China(type):
    __metaclass__ = MetaMetaclass

class Taiwan(type):
    __metaclass__ = MetaMetaclass

class A(object):
    __metaclass__ = China

class B(object):
    __metaclass__ = Taiwan

print A._label # Made in China
print B._label # Made in Taiwan

Edit

This is a pretty old question, but it's still getting upvotes, so I thought I'd add a link to a more comprehensive answer. If you'd like to read more about metaclasses and their uses, I've just published an article about it here.

The purpose of metaclasses isn't to replace the class/object distinction with metaclass/class - it's to change the behaviour of class definitions (and thus their instances) in some way. Effectively it's to alter the behaviour of the class statement in ways that may be more useful for your particular domain than the default. The things I have used them for are:

• Tracking subclasses, usually to register handlers. This is handy when using a plugin style setup, where you wish to register a handler for a particular thing simply by subclassing and setting up a few class attributes. eg. suppose you write a handler for various music formats, where each class implements appropriate methods (play / get tags etc) for its type. Adding a handler for a new type becomes:

  class Mp3File(MusicFile):
      extensions = ['.mp3']  # Register this type as a handler for mp3 files
      ...
      # Implementation of mp3 methods go here
  

  The metaclass then maintains a dictionary of {'.mp3' : MP3File, ... } etc, and constructs an object of the appropriate type when you request a handler through a factory function.

• Changing behaviour. You may want to attach a special meaning to certain attributes, resulting in altered behaviour when they are present. For example, you may want to look for methods with the name _get_foo and _set_foo and transparently convert them to properties. As a real-world example, here's a recipe I wrote to give more C-like struct definitions. The metaclass is used to convert the declared items into a struct format string, handling inheritance etc, and produce a class capable of dealing with it.

  For other real-world examples, take a look at various ORMs, like sqlalchemy's ORM or sqlobject. Again, the purpose is to interpret defintions (here SQL column definitions) with a particular meaning.

I have a class that handles non-interactive plotting, as a frontend to Matplotlib. However, on occasion one wants to do interactive plotting. With only a couple functions I found that I was able to increment the figure count, call draw manually, etc, but I needed to do these before and after every plotting call. So to create both an interactive plotting wrapper and an offscreen plotting wrapper, I found it was more efficient to do this via metaclasses, wrapping the appropriate methods, than to do something like:

class PlottingInteractive:
    add_slice = wrap_pylab_newplot(add_slice)

This method doesn't keep up with API changes and so on, but one that iterates over the class attributes in __init__ before re-setting the class attributes is more efficient and keeps things up to date:

class _Interactify(type):
    def __init__(cls, name, bases, d):
        super(_Interactify, cls).__init__(name, bases, d)
        for base in bases:
            for attrname in dir(base):
                if attrname in d: continue # If overridden, don't reset
                attr = getattr(cls, attrname)
                if type(attr) == types.MethodType:
                    if attrname.startswith("add_"):
                        setattr(cls, attrname, wrap_pylab_newplot(attr))
                    elif attrname.startswith("set_"):
                        setattr(cls, attrname, wrap_pylab_show(attr))

Of course, there might be better ways to do this, but I've found this to be effective. Of course, this could also be done in __new__ or __init__, but this was the solution I found the most straightforward.

Let's start with Tim Peter's classic quote:

Metaclasses are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don't (the people who actually need them know with certainty that they need them, and don't need an explanation about why). Tim Peters (c.l.p post 2002-12-22)

Having said that, I have (periodically) run across true uses of metaclasses. The one that comes to mind is in Django where all of your models inherit from models.Model. models.Model, in turn, does some serious magic to wrap your DB models with Django's ORM goodness. That magic happens by way of metaclasses. It creates all manner of exception classes, manager classes, etc. etc.

See django/db/models/base.py, class ModelBase() for the beginning of the story.

A reasonable pattern of metaclass use is doing something once when a class is defined rather than repeatedly whenever the same class is instantiated.

When multiple classes share the same special behaviour, repeating __metaclass__=X is obviously better than repeating the special purpose code and/or introducing ad-hoc shared superclasses.

But even with only one special class and no foreseeable extension, __new__ and __init__ of a metaclass are a cleaner way to initialize class variables or other global data than intermixing special-purpose code and normal def and class statements in the class definition body.

I was thinking the same thing just yesterday and completely agree. The complications in the code caused by attempts to make it more declarative generally make the codebase harder to maintain, harder to read and less pythonic in my opinion. It also normally requires a lot of copy.copy()ing (to maintain inheritance and to copy from class to instance) and means you have to look in many places to see whats going on (always looking from metaclass up) which goes against the python grain also. I have been picking through formencode and sqlalchemy code to see if such a declarative style was worth it and its clearly not. Such style should be left to descriptors (such as property and methods) and immutable data. Ruby has better support for such declarative styles and I am glad the core python language is not going down that route.

I can see their use for debugging, add a metaclass to all your base classes to get richer info. I also see their use only in (very) large projects to get rid of some boilerplate code (but at the loss of clarity). sqlalchemy for example does use them elsewhere, to add a particular custom method to all subclasses based on an attribute value in their class definition e.g a toy example

class test(baseclass_with_metaclass):
    method_maker_value = "hello"

could have a metaclass that generated a method in that class with special properties based on "hello" (say a method that added "hello" to the end of a string). It could be good for maintainability to make sure you did not have to write a method in every subclass you make instead all you have to define is method_maker_value.

The need for this is so rare though and only cuts down on a bit of typing that its not really worth considering unless you have a large enough codebase.