Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Speed performance difference between flattened dictionary and nested dictionary

I've developed a program in python, organizing data in flattened dictionaries. As the size of the dict increases, the program becomes slower due to a intensive keys search. Looking at nested dictionaries structure, seems to me that a "hierarchical" approach may speed up the search of the keys. Am I wrong?

Is the nested dict:

nested_dict = { 'dictA': {'key_1': 'value_1', 'key_2': 'value_2'},
                'dictB': {'key_3': 'value_3', 'key_4': 'value_4', 'key_5': 'value_5'},
                ...
                'dictZ': {'key_m': 'value_m', 'key_n': 'value_n'}}

faster than the flattened dict:

dictionary = {'key_1': 'value_1',
              'key_2': 'value_2',
              ...
              'key_n': 'value_n'}

Edit: Added few code examples

Below a piece of the code generally I use. The Program is quite big, so there isn't a specific code to be evaluated

Assignment:

dictionary['key_1'] = dictionary2['key_a']
dictionary['key_3'] = dictionary2['key_a']*dictionary['key_4']

Conditional statement:

if( (0 == dictionary['key_1']) and
    (dictionary2['key_b'] >= dictionary['key_3']) ):
like image 984
Nem Avatar asked Sep 27 '26 01:09

Nem


1 Answers

Looking at nested dictionaries structure, seems to me that a "hierarchical" approach may speed up the search of the keys. Am I wrong?

Yes :-)

The flat dict space has O(1) lookup irrespective of size. That is what makes hash tables so attractive as a data structure.

Adding hierarchy just adds extra hashing steps and lookup steps.

In some contexts, containers do get some cache locality benefits by being small, but in Python the containers have references to objects that are scattered all over memory, so compactness doesn't help much.

In addition, Python is an interpreted language, so adding a extra layer of lookups also entails more opcode evaluations. This would swamp any possible benefits for compactness.

like image 96
Raymond Hettinger Avatar answered Sep 28 '26 15:09

Raymond Hettinger



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!