python: cooperative supercall of __getattr__

Question

I'm working with somethign similar to this code:

class BaseClass(object):
    def __getattr__(self, attr):
        return lambda:'1'

class SubClass(BaseClass):
    def foo(self):
        suffix = '2'
        return super(SubClass, self).foo() + suffix

class SubClass2(SubClass):
    def foo(self):
        suffix = '3'
        return super(SubClass2, self).foo() + suffix

o = SubClass2()
print o.foo()

I'd expect to see output of '123', but I instead I get an error AttributeError: 'super' object has no attribute 'foo'. Python isn't even attempting to use the base class's __getattr__.

Without modifying the base class, and keeping the two super calls similar, I'm not able to get the output I want. Is there any cooperative supercall pattern that will work for me here?

I understand that super() overrides getattr in some way to do what it needs to do, but I'm asking if there's any reasonable workaround that allows a subclass's __getattr__ to be called when appropriate.

Answer 1

Ah, this is a great question!

In short, what's going on here is that the CPython internals occasionally take shortcuts when they are doing attribute lookups, and this kind of surprising behaviour is one of the consequences (another one being improved performance).

To understand exactly what's happening in this situation, we need to venture into the definition of super: http://hg.python.org/cpython/file/c24941251473/Objects/typeobject.c#l6689

Notice specifically that it doesn't define tp_getattr (aka __getattr__), but does define tp_getattro (aka __getattribute__):

PyTypeObject PySuper_Type = {
    PyVarObject_HEAD_INIT(&PyType_Type, 0)
    "super",                                    /* tp_name */
    ...
    0,                                          /* tp_getattr */
    ...
    super_getattro,                             /* tp_getattro */
    ...
};

(recall that __getattribute__ is called every time an attribute is requested, as opposed to __getattr__, which is only called if the attribute doesn't exist on the object (roughly: if the attribute isn't in the object's __dict__)).

Next, looking into the definition of super_getattro (aka super.__getattribute__), we can see the implementation is approximately:

class super(object):
    def __init__(self, obj_type, obj):
        self.obj_type = obj_type
        self.obj = obj

    def __getattribute__(self, attr):
        i = self.obj_type.__mro__.find(self.obj_type)
        i += 1
        while i < len(obj_type.__mro__):
            cur_type = self.obj_type.__mro__[i]
            cur_dict = cur_type.__dict___
            res = cur_dict.get(attr)
            if res is not None:
                return res
            i += 1
        return object.__getattribute__(self, attr)

Which makes it obvious why super doesn't play well with __getattr__ — super is only checking for attributes in the parent class' __dict__!

Fun aside: it seems like pypy (as of 2.1.0) behaves the same way:

$ pypy super.py 
Traceback (most recent call last):
  File "app_main.py", line 72, in run_toplevel
  File "super.py", line 16, in <module>
    print o.foo()
  File "super.py", line 13, in foo
    return super(SubClass2, self).foo() + suffix
  File "super.py", line 8, in foo
    return super(SubClass, self).foo() + suffix
AttributeError: 'super' object has no attribute 'foo'

Answer 2

This seems to work correctly. I don't currently see why the standard super class doesn't do this.

class super2(super):
    def __getattr__(self, attr):
        return self.__self__.__getattr__(attr)

class BaseClass(object):
    def __getattr__(self, attr):
        return lambda:'1'

class SubClass(BaseClass):
    def foo(self):
        suffix = '2'
        return super2(SubClass, self).foo() + suffix

class SubClass2(SubClass):
    def foo(self):
        suffix = '3'
        return super2(SubClass2, self).foo() + suffix

o = SubClass2()
print o.foo()