Python pickle crash when trying to return default value in __getattr__

Question

I have a dictionary like class that I use to store some values as attributes. I recently added some logic(__getattr__) to return None if an attribute doesn't exist. As soon as I did this pickle crashed, and I wanted some insight into why?

Test Code:

import cPickle
class DictionaryLike(object):
    def __init__(self, **kwargs):
        self.__dict__.update(kwargs)

    def __iter__(self):
        return iter(self.__dict__)

    def __getitem__(self, key):
        if(self.__dict__.has_key(key)):
            return self.__dict__[key]
        else:
            return None

    ''' This is the culprit...'''    
    def __getattr__(self, key):
        print 'Retreiving Value ' , key
        return self.__getitem__(key)

class SomeClass(object):
    def __init__(self, kwargs={}):
       self.args = DictionaryLike(**kwargs)


someClass = SomeClass()
content = cPickle.dumps(someClass,-1)
print content

---

Result:

Retreiving Value  __getnewargs__
Traceback (most recent call last):
  File <<file>> line 29, in <module>
    content = cPickle.dumps(someClass,-1)
TypeError: 'NoneType' object is not callable`

---

Did I do something stupid? I had read a post that deepcopy() might require that I throw an exception if a key doesn't exist? If this is the case is there any easy way to achieve what I want without throwing an exception?

End result is that if some calls

someClass.args.i_dont_exist

I want it to return None.

Answer 1

Implementing __getattr__ is a bit tricky, since it is called for every non-existing attribute. In your case, the pickle module tests your class for the __getnewargs__ special method and receives None, which is obviously not callable.

You might want to alter __getattr__ to call the base implementation for magic names:

def __getattr__(self, key):
    if key.startswith('__') and key.endswith('__'):
        return super(DictionaryLike, self).__getattr__(key)
    return self.__getitem__(key)

I usually pass through all names starting with an underscore, so that I can sidestep the magic for internal symbols.

Answer 2

You need to raise an AttributeError when an attribute is not present in your class:

def __getattr__(self, key):
    i = self.__getitem__(key)
    if i == None:
        raise AttributeError
    return self.__getitem__(key)

I am going to assume that this behavior is required. From the python documentation for getattr, "Called when an attribute lookup has not found the attribute in the usual places (i.e. it is not an instance attribute nor is it found in the class tree for self). name is the attribute name. This method should return the (computed) attribute value or raise an AttributeError exception."

There is no way to tell pickle etc that the attribute it's looking for is not found unless you raise the exception. For example, in your error message pickle is looking for a special callable method called __getnewargs__, pickle expects that if the AttributeError exception is not found the return value is callable.

I guess one potential work around you could perhaps try defining all of the special methods pickle is looking for as dummy methods?