Python / YAML: How to initialize additional objects not just from the YAML file, within loadConfig?

Question

I have what I think is a small misconception with loading some YAML objects. I defined the class below.

What I want to do is load some objects with the overridden loadConfig function for YAMLObjects. Some of these come from my .yaml file, but others should be built out of objects loaded from the YAML file.

For instance, in the class below, I load a member object named "keep" which is a string naming some items to keep in the region. But I want to also parse this into a list and have the list stored as a member object too. And I don't want the user to have to give both the string and list version of this parameter in the YAML.

My current work around has been to override the __getattr__ function inside Region and make it create the defaults if it looks and doesn't find them. But this is clunky and more complicated than needed for just initializing objects.

What convention am I misunderstanding here. Why doesn't the loadConfig method create additional things not found in the YAML?

import yaml, pdb

class Region(yaml.YAMLObject):
    yaml_tag = u'!Region'

    def __init__(self, name, keep, drop):
        self.name = name
        self.keep = keep
        self.drop = drop

        self.keep_list = self.keep.split("+")
        self.drop_list = self.drop.split("+")
        self.pattern = "+".join(self.keep_list) + "-" + "-".join(self.drop_list)
    ###

    def loadConfig(self, yamlConfig):
        yml = yaml.load_all(file(yamlConfig))
        for data in yml:

            # These get created fine
            self.name = data["name"]
            self.keep = data["keep"]
            self.drop = data["drop"]

            # These do not get created.
            self.keep_list = self.keep.split("+")
            self.drop_list = self.drop.split("+")
            self.pattern = "+".join(self.keep_list) + "-" + "-".join(self.drop_list)
    ###  
### End Region

if __name__ == "__main__":
    my_yaml = "/home/path/to/test.yaml"
    region_iterator = yaml.load_all(file(my_yaml))

    # Set a debug breakpoint to play with region_iterator and
    # confirm the extra stuff isn't created.
    pdb.set_trace()

And here is test.yaml so you can run all of this and see what I mean:

 Regions:

   # Note: the string conventions below are for an
   # existing system. This is a shortened, representative
   # example.

   Market1:  
    !Region                 
        name: USAndGB
        keep: US+GB
        drop: !!null 

   Market2:
    !Region
        name: CanadaAndAustralia
        keep: CA+AU
        drop: !!null 

And here, for example, is what it looks like for me when I run this in an IPython shell and explore the loaded object:

In [57]: %run "/home/espears/testWorkspace/testRegions.py"
--Return--
> /home/espears/testWorkspace/testRegions.py(38)<module>()->None
-> pdb.set_trace()
(Pdb) region_iterator
<generator object load_all at 0x1139d820>
(Pdb) tmp = region_iterator.next()
(Pdb) tmp
{'Regions': {'Market2': <__main__.Region object at 0x1f858550>, 'Market1': <__main__.Region object at 0x11a91e50>}}
(Pdb) us = tmp['Regions']['Market1']
(Pdb) us
<__main__.Region object at 0x11a91e50>
(Pdb) us.name
'USAndGB'
(Pdb) us.keep
'US+GB'
(Pdb) us.keep_list
*** AttributeError: 'Region' object has no attribute 'keep_list'

Answer 1

A pattern I have found useful for working with yaml for classes that are basically storage is to have the loader use the constructor so that objects are created in the same way as when you make them normally. If I understand what you are attempting to do correctly, this kind of structure might be useful:

import inspect
import yaml
from collections import OrderedDict

class Serializable(yaml.YAMLObject):
    __metaclass__ = yaml.YAMLObjectMetaclass
    @property
    def _dict(self):
        dump_dict = OrderedDict()

        for var in inspect.getargspec(self.__init__).args[1:]:
            if getattr(self, var, None) is not None:
                item = getattr(self, var)
                if isinstance(item, np.ndarray) and item.ndim == 1:
                    item = list(item)
                dump_dict[var] = item

        return dump_dict

    @classmethod
    def to_yaml(cls, dumper, data):
        return ordered_dump(dumper, '!{0}'.format(data.__class__.__name__), 
                            data._dict)


    @classmethod
    def from_yaml(cls, loader, node):
        fields = loader.construct_mapping(node, deep=True)
        return cls(**fields)

def ordered_dump(dumper, tag, data):
    value = []
    node = yaml.nodes.MappingNode(tag, value)
    for key, item in data.iteritems():
        node_key = dumper.represent_data(key)
        node_value = dumper.represent_data(item)
        value.append((node_key, node_value))

    return node

You would then want to have your Region class inherit from Serializable, and remove the loadConfig stuff. The code I posted inspects the constructor to see what data to save to the yaml file, and then when loading a yaml file calls the constructor with that same set of data. That way you just have to get the logic right in your constructor and the yaml loading should get it for free.

That code was ripped from one of my projects, apologies in advance if it doesn't quite work. It is also slightly more complicated than it needs to be because I wanted to control the order of output by using OrderedDict. You could replace my ordered_dump function with a call to dumper.represent_dict.