Using __getattr__ and meeting expected behaviour for subclasses

Question

I am the author -- and currently almost definitely the sole user, if on multiple projects -- of a simple database layer (inspired by minimongo) for MongoDB called kale. My current use of __getattr__ in the base class for models has led to some hard-to-track bugs.

The problem I've run into was articulated concisely last June by David Halter on this site. The discussion is interesting, however no solutions were offered.

In short:

>>> class A(object):
...     @property
...     def a(self):
...         print "We're here -> attribute lookup found 'a' in one of the usual places!"
...         raise AttributeError
...         return "a"
...     
...     def __getattr__(self, name):
...         print "We're here -> attribute lookup has not found the attribute in the usual places!"
...         print('attr: ', name)
...         return "not a"
... 
>>> print(A().a)
We're here -> attribute lookup found 'a' in one of the usual places!
We're here -> attribute lookup has not found the attribute in the usual places!
('attr: ', 'a')
not a
>>>

Note that this contradictory behaviour is not what I would expect from reading the official python documentation:

object.__getattr__(self, name)

Called when an attribute lookup has not found the attribute in the usual places (i.e. it is not an instance attribute nor is it found in the class tree for self). name is the attribute name.

(It would be nice if they mentioned that AttributeError is the means by which "attribute lookup" knows if an attribute has been found in "the usual places" or not. The clarifying parenthetical seems to me at best incomplete.)

In practice, this has caused problems tracking down bugs caused by programming errors wherever an AttributeError is raised in a @property descriptor thing.

>>> class MessedAttrMesser(object):
...     things = {
...         'one': 0,
...         'two': 1,
...     }
...     
...     def __getattr__(self, attr):
...         try:
...             return self.things[attr]
...         except KeyError as e:
...             raise AttributeError(e)
...     
...     @property
...     def get_thing_three(self):
...         return self.three
... 
>>> 
>>> blah = MessedAttrMesser()
>>> print(blah.one)
0
>>> print(blah.two)
1
>>> print(blah.get_thing_three)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 11, in __getattr__
AttributeError: 'get_thing_three'
>>>

In this case it's pretty obvious what's going on by inspecting the class as a whole. However, if you rely on the message from the stack trace, AttributeError: 'get_thing_three', it makes no sense, since, clearly, get_thing_three looks to be as valid an attribute as they come.

The purpose of kale is to provide a base class for building models. As such, the base model code is hidden away from the end programmer, and masking the cause of errors like this is not ideal.

The end programmer (cough me) may choose to use @property descriptors on their models, and their code should work and fail in the ways they would expect.

The Question

How can I allow AttributeErrors to propagate through my base class which has defined __getattr__?

Answer 1

Basically, you can't --- or at least, not in simple and foolproof way. As you noted, AttributeError is the mechanism Python uses to determine whether an attribute is "found in the usual places". Although the __getattr__ documentation doesn't mention this, it is made somewhat more clear in the documentation for __getattribute__, as described in this answer to the question you already linked to.

You could override __getattribute__ and catch AttributeError there, but if you caught one, you'd have no obvious way to tell whether it was a "real" error (meaning the attribute really wasn't found), or an error raised by user code during the lookup process. In theory you could inspect the traceback looking for particular things, or do various other hacks to try to verify the attribute's existence, but these approaches are going to be more fragile and dangerous than the existing behavior.

One other possibility is to write your own descriptor that is based on property, but catches AttributeErrors and re-raises them as something else. This would, however, require you to use this property-replacement instead of the builtin property. In addition, it would mean that AttributeErrors raised from inside descriptor methods are not propagated as AttributeErrors but as something else (whatever you replace them with). Here is an example:

class MyProp(property):
    def __get__(self, obj, cls):
        try:
            return super(MyProp, self).__get__(obj, cls)
        except AttributeError:
            raise ValueError, "Property raised AttributeError"

class A(object):
    @MyProp
    def a(self):
        print "We're here -> attribute lookup found 'a' in one of the usual places!"
        raise AttributeError
        return "a"

    def __getattr__(self, name):
        print "We're here -> attribute lookup has not found the attribute in the usual places!"
        print('attr: ', name)
        return "not a"

>>> A().a
We're here -> attribute lookup found 'a' in one of the usual places!
Traceback (most recent call last):
  File "<pyshell#8>", line 1, in <module>
    A().a
  File "<pyshell#6>", line 6, in __get__
    raise ValueError, "Property raised AttributeError"
ValueError: Property raised AttributeError

Here AttributeErrors are replaced with ValueErrors. This could be fine if all you want is to make sure the exception "breaks out" of the attribute-access mechanism and can propagate up to the next level. But if you have complex exception-catching code that is expecting to see an AttributeError, it will miss this error, since the exception type has changed.

(Also, this example obviously only deals with property getters, not setters, but it should be clear how to extend the idea.)

I guess as an extension of this solution, you could combine this MyProp idea with a custom __getattribute__. Basically, you could define a custom exception class, say PropertyAttributeError, and have the property-replacement reraise AttributeError as PropertyAttributeError. Then, in your custom __getattribute__, you could catch PropertyAttributeError and re-raise it as AttributeError. Basically the MyProp and the __getattribute__ could act as a "shunt" that bypasses Python's normal handling by converting the error from AttributeError to something else, and then converting it back to AttributeError again once it's "safe" to do so. However, my feeling is it's not worth it to do this, as __getattribute__ can have a substantial performance impact.

One little addendum: a bug has been raised about this on the Python bug tracker, and there is recent activity on possible solutions, so it's possible it may be fixed in future versions.

Answer 2

What happens in your code:

first case with class A:

>>>print(A().a)

1. create an instance of A
2. access an attribute called 'A' on the instance

now python, following its data model, trys to find A.a using object.__getattribute__(since you haven't provided your custom __getattribute__)

however :

@property
def a(self):
    print "We're here -> attribute lookup found 'a' in one of the usual places!"
    raise AttributeError # <= an AttributeError is raised - now python resorts to '__getattr__'
    return "a" # <= this code is unreachable

so, since __getattribute__ lookup ended with AttributeError, it calls your __getattr__ :

    def __getattr__(self, name):
...         print "We're here -> attribute lookup has not found the attribute in the usual places!"
...         print('attr: ', name)
...         return "not a" #it returns 'not a'

what happens in your second code:

you access blah.get_thing_three by __getattribute__. since get_thing_three fails(no three in things) with an AttributeError, now your __getattr__ tries to lookup get_thing_three in things, which also fails - you get exception for get_thing_three because it has higher precedence.

what you can do :

you have to write custom __getattribute__ and __getattr__ both. but, in most cases it won't get you far, and other people using your code won't expect some custom data protocols.

well, I have an advice for you(I have written a crude mongodb orm that I use internally) : don't rely on your __getattr__ inside your document-to-object mapper. use direct access to document inside your class (I think it doesn't hurt any encapsulation). here's my example:

class Model(object):
  _document = { 'a' : 1, 'b' : 2 }
  def __getattr__(self, name): 
     r"""syntactic sugar for those who are using this class externally. 
     >>>foo = Model()
     >>>foo.a
     1"""

  @property
  def ab_sum(self):
     try :
        return self._document[a] + self._document[b]
     except KeyError:
        raise #something that isn't AttributeError