How should python dictionaries be stored in pytables?

Question

pytables doesn't natively support python dictionaries. The way I've approached it is to make a data structure of the form:

tables_dict = {
'key'         : tables.StringCol(itemsize=40),
'value'       : tables.Int32Col(),
}

(note that I ensure that the keys are <40 characters long) and then create a table using this structure:

file_handle.createTable('/', 'dictionary', tables_dict)

and then populate it with:

file_handle.dictionary.append(dictionary.items())

and retrieve data with:

dict(file_handle.dictionary.read())

This works ok, but reading the dictionary back in is extremely slow. I think the problem is that the read() function is causing the entire dictionary to be loaded into memory, which shouldn't really be necessary. Is there a better way to do this?

Answer 1

You can ask PyTables to search inside the table, and also create an index on the key column to speed that up.

To create an index:

table.cols.key.createIndex()

To query the values where key equals the variable search_key:

[row['value'] for row in table.where('key == search_key')]

http://pytables.github.com/usersguide/optimization.html#searchoptim