How to store numerical lookup table in Python (with labels)

Question

I have a scientific model which I am running in Python which produces a lookup table as output. That is, it produces a many-dimensional 'table' where each dimension is a parameter in the model and the value in each cell is the output of the model.

My question is how best to store this lookup table in Python. I am running the model in a loop over every possible parameter combination (using the fantastic itertools.product function), but I can't work out how best to store the outputs.

It would seem sensible to simply store the output as a ndarray, but I'd really like to be able to access the outputs based on the parameter values not just indices. For example, rather than accessing the values as table[16][5][17][14] I'd prefer to access them somehow using variable names/values, for example:

table[solar_z=45, solar_a=170, type=17, reflectance=0.37]

or something similar to that. It'd be brilliant if I were able to iterate over the values and get their parameter values back - that is, being able to find out that table[16]... corresponds to the outputs for solar_z = 45.

Is there a sensible way to do this in Python?

Answer 1

Why don't you use a database? I have found MongoDB (and the official Python driver, Pymongo) to be a wonderful tool for scientific computing. Here are some advantages:

• Easy to install - simply download the executables for your platform (2 minutes tops, seriously).
• Schema-less data model
• Blazing fast
• Provides map/reduce functionality
• Very good querying functionalities

So, you could store each entry as a MongoDB entry, for example:

{"_id":"run_unique_identifier",
 "param1":"val1",
 "param2":"val2" # etcetera
}

Then you could query the entries as you will:

import pymongo
data = pymongo.Connection("localhost", 27017)["mydb"]["mycollection"]
for entry in data.find(): # this will yield all results
 yield entry["param1"] # do something with param1

Whether or not MongoDB/pymongo are the answer to your specific question, I don't know. However, you could really benefit from checking them out if you are into data-intensive scientific computing.