Storage capacity of dictionary and want to store huge data into dictionary in python

Question

I want to store huge data into a dictionary in python. Huge data may be around 21 GB. I wrote a snippet to do so. Storing the integer values inside the dictionary.

Code:

import timeit
import sys
dicts = {}
print "\n***Building dict..."
start = timeit.default_timer()
for j in range(0,5):
    for i in range(0,1000000):
        dicts[''+str(j)+str(i)] = i
        print str(i) + '-' + str(j)
        print "Size : ", sys.getsizeof(dicts)/1024/1024, " MB"
print "Total time of build dict", timeit.default_timer() - start 

During runtime, when I reached the size using **getsizeof(dicts)** around 1.2GB, it fails to store the values in dictionary but doesn't show any error. Does Dictionary has some capacity to store the data?

So, the question is how can I store huge data into the dictionary?

NOTE: Doesn't require to store the data in files or databases. because I want to retrieve the key, value pair very fast.

Answer 1

The limit of the size of a Python dict depends on the free memory the OS makes available. The problem is that as a dict grows (resizes) it has to copy itself to redistribute the keys so when the dictionary get really huge this process can start to require a lot more memory than is actually available.