Python: Idiomatic properties for structured data?

Question

I've got a bad smell in my code. Perhaps I just need to let it air out for a bit, but right now it's bugging me.

I need to create three different input files to run three Radiative Transfer Modeling (RTM) applications, so that I can compare their outputs. This process will be repeated for thousands of sets of inputs, so I'm automating it with a python script.

I'd like to store the input parameters as a generic python object that I can pass to three other functions, who will each translate that general object into the specific parameters needed to run the RTM software they are responsible. I think this makes sense, but feel free to criticize my approach.

There are many possible input parameters for each piece of RTM software. Many of them over-lap. Most of them are kept at sensible defaults, but should be easily changed.

I started with a simple dict

config = {
    day_of_year: 138,
    time_of_day: 36000, #seconds
    solar_azimuth_angle: 73, #degrees
    solar_zenith_angle: 17, #degrees
    ...
}

There are a lot of parameters, and they can be cleanly categorized into groups, so I thought of using dicts within the dict:

config = {
    day_of_year: 138,
    time_of_day: 36000, #seconds
    solar: {
        azimuth_angle: 73, #degrees
        zenith_angle: 17, #degrees
        ...
    },
    ...
}

I like that. But there are a lot of redundant properties. The solar azimuth and zenith angles, for example, can be found if the other is known, so why hard-code both? So I started looking into python's builtin property. That lets me do nifty things with the data if I store it as object attributes:

class Configuration(object):
    day_of_year = 138,
    time_of_day = 36000, #seconds
    solar_azimuth_angle = 73, #degrees
    @property
    def solar_zenith_angle(self):
        return 90 - self.solar_azimuth_angle
    ...

config = Configuration()

But now I've lost the structure I had from the second dict example.

Note that some of the properties are less trivial than my solar_zenith_angle example, and might require access to other attributes outside of the group of attributes it is a part of. For example I can calculate solar_azimuth_angle if I know the day of year, time of day, latitude, and longitude.

What I'm looking for:

A simple way to store configuration data whose values can all be accessed in a uniform way, are nicely structured, and may exist either as attributes (real values) or properties (calculated from other attributes).

A possibility that is kind of boring:

Store everything in the dict of dicts I outlined earlier, and having other functions run over the object and calculate the calculatable values? This doesn't sound fun. Or clean. To me it sounds messy and frustrating.

An ugly one that works:

After a long time trying different strategies and mostly getting no where, I came up with one possible solution that seems to work:

My classes: (smells a bit func-y, er, funky. def-initely.)

class SubConfig(object):
    """
    Store logical groupings of object attributes and properties.

    The parent object must be passed to the constructor so that we can still
    access the parent object's other attributes and properties. Useful if we
    want to use them to compute a property in here.
    """
    def __init__(self, parent, *args, **kwargs):
        super(SubConfig, self).__init__(*args, **kwargs)
        self.parent = parent


class Configuration(object):
    """
    Some object which holds many attributes and properties.

    Related configurations settings are grouped in SubConfig objects.
    """
    def __init__(self, *args, **kwargs):
        super(Configuration, self).__init__(*args, **kwargs)
        self.root_config = 2

        class _AConfigGroup(SubConfig):
            sub_config = 3
            @property
            def sub_property(self):
                return self.sub_config * self.parent.root_config
        self.group = _AConfigGroup(self) # Stinky?!

How I can use them: (works as I would like)

config = Configuration()

# Inspect the state of the attributes and properties.
print("\nInitial configuration state:")
print("config.rootconfig: %s" % config.root_config)
print("config.group.sub_config: %s" % config.group.sub_config)
print("config.group.sub_property: %s (calculated)" % config.group.sub_property)

# Inspect whether the properties compute the correct value after we alter
# some attributes.
config.root_config = 4
config.group.sub_config = 5

print("\nState after modifications:")
print("config.rootconfig: %s" % config.root_config)
print("config.group.sub_config: %s" % config.group.sub_config)
print("config.group.sub_property: %s (calculated)" % config.group.sub_property)

The behavior: (output of execution of all of the above code, as expected)

Initial configuration state:
config.rootconfig: 2
config.group.sub_config: 3
config.group.sub_property: 6 (calculated)

State after modifications:
config.rootconfig: 4
config.group.sub_config: 5
config.group.sub_property: 20 (calculated)

Why I don't like it:

Storing configuration data in class definitions inside of the main object's __init__() doesn't feel elegant. Especially having to instantiate them immediately after definition like that. Ugh. I can deal with that for the parent class, sure, but doing it in a constructor...

Storing the same classes outside the main Configuration object doesn't feel elegant either, since properties in the inner classes may depend on the attributes of Configuration (or their siblings inside it).

I could deal with defining the functions outside of everything, so inside having things like

@property
def solar_zenith_angle(self):
   return calculate_zenith(self.solar_azimuth_angle)

but I can't figure out how to do something like

@property
def solar.zenith_angle(self):
    return calculate_zenith(self.solar.azimuth_angle)

(when I try to be clever about it I always run into <property object at 0xXXXXX>)

So what is the right way to go about this? Am I missing something basic or taking a very wrong approach? Does anyone know a clever solution?

Help! My python code isn't beautiful! I must be doing something wrong!

Answer 1

Phil,

Your hesitation about func-y config is very familiar to me :)

I suggest you to store your config not as a python file but as a structured data file. I personally prefer YAML because it looks clean, just as you designed in the very beginning. Of course, you will need to provide formulas for the auto calculated properties, but it is not too bad unless you put too much code. Here is my implementation using PyYAML lib.

The config file (config.yml):

day_of_year: 138
time_of_day: 36000 # seconds
solar:
  azimuth_angle: 73 # degrees
  zenith_angle: !property 90 - self.azimuth_angle

The code:

import yaml

yaml.add_constructor("tag:yaml.org,2002:map", lambda loader, node:
    type("Config", (object,), loader.construct_mapping(node))())

yaml.add_constructor("!property", lambda loader, node:
    property(eval("lambda self: " + loader.construct_scalar(node))))

config = yaml.load(open("config.yml"))

print "LOADED config.yml"
print "config.day_of_year:", config.day_of_year
print "config.time_of_day:", config.time_of_day
print "config.solar.azimuth_angle:", config.solar.azimuth_angle
print "config.solar.zenith_angle:", config.solar.zenith_angle, "(calculated)"
print

config.solar.azimuth_angle = 65
print "CHANGED config.solar.azimuth_angle = 65"
print "config.solar.zenith_angle:", config.solar.zenith_angle, "(calculated)"

The output:

LOADED config.yml
config.day_of_year: 138
config.time_of_day: 36000
config.solar.azimuth_angle: 73
config.solar.zenith_angle: 17 (calculated)

CHANGED config.solar.azimuth_angle = 65
config.solar.zenith_angle: 25 (calculated)

The config can be of any depth and properties can use any subgroup values. Try this for example:

a: 1
b:
  c: 3
  d: some text
  e: true
  f:
    g: 7.01
x: !property self.a + self.b.c + self.b.f.g

Assuming you already loaded this config:

>>> config
<__main__.Config object at 0xbd0d50>
>>> config.a
1
>>> config.b
<__main__.Config object at 0xbd3bd0>
>>> config.b.c
3
>>> config.b.d
'some text'
>>> config.b.e
True
>>> config.b.f
<__main__.Config object at 0xbd3c90>
>>> config.b.f.g
7.01
>>> config.x
11.01
>>> config.b.f.g = 1000
>>> config.x
1004

UPDATE

Let us have a property config.b.x which uses both self, parent and subgroup attributes in its formula:

a: 1
b:
  x: !property self.parent.a + self.c + self.d.e
  c: 3
  d:
    e: 5

Then we just need to add a reference to parent in subgroups:

import yaml

def construct_config(loader, node):
    attrs = loader.construct_mapping(node)
    config = type("Config", (object,), attrs)()
    for k, v in attrs.iteritems():
        if v.__class__.__name__ == "Config":
            setattr(v, "parent", config)
    return config

yaml.add_constructor("tag:yaml.org,2002:map", construct_config)

yaml.add_constructor("!property", lambda loader, node:
    property(eval("lambda self: " + loader.construct_scalar(node))))

config = yaml.load(open("config.yml"))

And let's see how it works:

>>> config.a
1
>>> config.b.c
3
>>> config.b.d.e
5
>>> config.b.parent == config
True
>>> config.b.d.parent == config.b
True
>>> config.b.x
9
>>> config.a = 1000
>>> config.b.x
1008