Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Maximum size of a dictionary in Python?

I'm using a dictionary to hold a large number of objects, and have a string name for each of them. To be specific, here is my code:

from itertools import product
for (i,j,k) in product(range(N),range(M),range(K)):
    var_name='x_'+'_'+str(i)+str(j)+'_'+str(k)
    var_dict[var_name] = f(var_name,other_params)
print len(var_dict)

f(...) returns an object. In my code N=363, M=500, and K=2. So I expect 363000 entries in the dictionary. But when I check the length of var_dict, it is 330860!

(Pdb) len(var_dict)
330860

Here are my questions:

  1. Is there any explanation for that? E.g. is there any limit for the number of items that built-in hash table of python can address?

  2. What can I do to solve this problem?

like image 467
Alt Avatar asked May 08 '14 03:05

Alt


1 Answers

The problem is here:

str(i)+str(j)

This does not produce unique identifiers. For example, the value set when i=1 and j=11 will be overwritten by the value set when i=11 and j=1 (there are many more instances as well).

You can fix the problem by inserting some delimiter character between the two numbers (such as an underscore like you have between j and k).

like image 176
nobody Avatar answered Oct 14 '22 08:10

nobody