Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python MemoryError on large array

Tags:

python

memory

This is the python script that I'm trying to run:

n = 50000000000 ##50 billion 
b = [0]*n
for x in range(0,n):
    b[x] = random.randint(1,899999)

... But the output I'm getting is:

E:\python\> python sort.py
Traceback (most recent call last):
  File "E:\python\sort.py", line 8, in <module>
    b = [0]*n
MemoryError

So, what do I do now?

like image 945
Adit Srivastava Avatar asked May 18 '17 19:05

Adit Srivastava


1 Answers

The size of the list you are generating (which is 50 billion not 5).

An int object instance takes 24 bytes (sys.getsizeof(int(899999)), the upper limit of your random numbers), so that list would take 50,000,000,000 * 24 bytes, which is about 1.09 TB.

In other words to create such a list you would need at least 1118 GB of RAM in your computer.

I don't know what your use case is, but you should consider a different approach to what you are trying to solve (maybe define a generator, or just don't store your numbers in memory and instead directly use the numbers in the for loop).

like image 162
Tenchi2xh Avatar answered Sep 17 '22 20:09

Tenchi2xh