Why does decorating a class break the descriptor protocol, thus preventing staticmethod objects from behaving as expected?

Question

I need a little bit of help understanding the subtleties of the descriptor protocol in Python, as it relates specifically to the behavior of staticmethod objects. I'll start with a trivial example, and then iteratively expand it, examining it's behavior at each step:

class Stub:
    @staticmethod
    def do_things():
        """Call this like Stub.do_things(), with no arguments or instance."""
        print "Doing things!"

At this point, this behaves as expected, but what's going on here is a bit subtle: When you call Stub.do_things(), you are not invoking do_things directly. Instead, Stub.do_things refers to a staticmethod instance, which has wrapped the function we want up inside it's own descriptor protocol such that you are actually invoking staticmethod.__get__, which first returns the function that we want, and then gets called afterwards.

>>> Stub
<class __main__.Stub at 0x...>
>>> Stub.do_things
<function do_things at 0x...>
>>> Stub.__dict__['do_things']
<staticmethod object at 0x...>
>>> Stub.do_things()
Doing things!

So far so good. Next, I need to wrap the class in a decorator that will be used to customize class instantiation -- the decorator will determine whether to allow new instantiations or provide cached instances:

def deco(cls):
    def factory(*args, **kwargs):
        # pretend there is some logic here determining
        # whether to make a new instance or not
        return cls(*args, **kwargs)
    return factory

@deco
class Stub:
    @staticmethod
    def do_things():
        """Call this like Stub.do_things(), with no arguments or instance."""
        print "Doing things!"

Now, naturally this part as-is would be expected to break staticmethods, because the class is now hidden behind it's decorator, ie, Stub not a class at all, but an instance of factory that is able to produce instances of Stub when you call it. Indeed:

>>> Stub
<function factory at 0x...>
>>> Stub.do_things
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'function' object has no attribute 'do_things'
>>> Stub()
<__main__.Stub instance at 0x...>
>>> Stub().do_things
<function do_things at 0x...>
>>> Stub().do_things()
Doing things!

So far I understand what's happening here. My goal is to restore the ability for staticmethods to function as you would expect them to, even though the class is wrapped. As luck would have it, the Python stdlib includes something called functools, which provides some tools just for this purpose, ie, making functions behave more like other functions that they wrap. So I change my decorator to look like this:

def deco(cls):
    @functools.wraps(cls)
    def factory(*args, **kwargs):
        # pretend there is some logic here determining
        # whether to make a new instance or not
        return cls(*args, **kwargs)
    return factory

Now, things start to get interesting:

>>> Stub
<function Stub at 0x...>
>>> Stub.do_things
<staticmethod object at 0x...>
>>> Stub.do_things()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'staticmethod' object is not callable
>>> Stub()
<__main__.Stub instance at 0x...>
>>> Stub().do_things
<function do_things at 0x...>
>>> Stub().do_things()
Doing things!

Wait.... what? functools copies the staticmethod over to the wrapping function, but it's not callable? Why not? What did I miss here?

I was playing around with this for a bit and I actually came up with my own reimplementation of staticmethod that allows it to function in this situation, but I don't really understand why it was necessary or if this is even the best solution to this problem. Here's the complete example:

class staticmethod(object):
    """Make @staticmethods play nice with decorated classes."""

    def __init__(self, func):
        self.func = func

    def __call__(self, *args, **kwargs):
        """Provide the expected behavior inside decorated classes."""
        return self.func(*args, **kwargs)

    def __get__(self, obj, objtype=None):
        """Re-implement the standard behavior for undecorated classes."""
        return self.func

def deco(cls):
    @functools.wraps(cls)
    def factory(*args, **kwargs):
        # pretend there is some logic here determining
        # whether to make a new instance or not
        return cls(*args, **kwargs)
    return factory

@deco
class Stub:
    @staticmethod
    def do_things():
        """Call this like Stub.do_things(), with no arguments or instance."""
        print "Doing things!"

Indeed it works exactly as expected:

>>> Stub
<function Stub at 0x...>
>>> Stub.do_things
<__main__.staticmethod object at 0x...>
>>> Stub.do_things()
Doing things!
>>> Stub()
<__main__.Stub instance at 0x...>
>>> Stub().do_things
<function do_things at 0x...>
>>> Stub().do_things()
Doing things!

What approach would you take to make a staticmethod behave as expected inside a decorated class? Is this the best way? Why doesn't the builtin staticmethod implement __call__ on it's own in order for this to just work without any fuss?

Thanks.

Answer 1

The problem is that you're changing the type of Stub from a class to a function. This is a pretty serious violation and it's not much surprise that things are breaking.

The technical reason that your staticmethods are breaking is that functools.wraps works by copying __name__, __doc__ and __module__ etc. (source: http://hg.python.org/cpython/file/3.2/Lib/functools.py) from the wrapped instance to the wrapper, while updating the wrapper's __dict__ from the wrapped instance's __dict__. It should be obvious now why staticmethod doesn't work - its descriptor protocol is being invoked on a function instead of a class, so it gives up on returning a bound callable and just returns its non-callable self.

w.r.t. actually doing what you're interested in (some kind of Singleton?), you probably want your decorator to return a class with a __new__ that has the required behaviour. You don't need to be worried about __init__ being called unwanted, as long as your wrapper class __new__ doesn't actually return a value of the wrapper class type, rather an instance of the wrapped class:

def deco(wrapped_cls):
    @functools.wraps(wrapped_cls)
    class Wrapper(wrapped_cls):
        def __new__(cls, *args, **kwargs):
            ...
            return wrapped_cls(*args, **kwargs)
    return Wrapper

Note the distinction between the wrapped_cls argument to the decorator (that becomes closed over in the wrapper class) and the cls argument to Wrapper.__new__.

Note that it's perfectly OK to use functools.wraps on a class wrapping a class - just not on a class wrapping a function!

You can also modify the wrapped class, in which case you don't need functools.wraps:

def deco(wrapped_cls):
    def __new__(cls, *args, **kwargs)
        ...
        return super(wrapped_cls, cls)(*args, **kwargs)
    wrapped_cls.__new__ = classmethod(__new__)
    return wrapped_cls

Note however that this method will end up invoking __init__ on existing instances, so you'll have to work around that (e.g. by wrapping __init__ to short-circuit on existing instances).

As an addendum: it might be possible to make your function-wrapping-a-class decorator work in the cases you know about with a lot of effort, but you'll still run into problems - for example, isinstance(myObject, Stub) has no chance of working as Stub is no longer a type!

Answer 2

you almost did what i yould have done:

def deco(cls):
    class factory(cls):
        def __new__(cls_factory, *args, **kwargs):
            # pretend there is some logic here determining
            # whether to make a new instance or not
            return cls.__new__(*args, **kwargs)
    return factory

that should do it. Problem may be that __init__ is called also on old instances returned by __new__.