How do I work with large, memory hungry numpy array?

Question

I have a program which creates an array:

List1 = zeros((x, y), dtype=complex_)

Currently I am using x = 500 and y = 1000000. I will initialize the first column of the list by some formula. Then the subsequent columns will calculate their own values based on the preceding column. After the list is completely filled, I will then display this multidimensional array using imshow().

The size of each value (item) in the list is 24 bytes. A sample value from the code is: 4.63829355451e-32

When I run the code with y = 10000000, it takes up too much RAM and the system stops the run. How do I solve this problem? Is there a way to save my RAM while still being able to process the list using imshow() easily? Also, how large a list can imshow() display?

Answer 1

There's no way to solve this problem (in any general way).

Computers (as commonly understood) have a limited amount of RAM, and they require elements to be in RAM in order to operate on them.

An complex128 array size of 10000000x500 would require around 74GiB to store. You'll need to somehow reduce the amount of data you're processing if you hope to use a regular computer to do it (as opposed to a supercomputer).

A common technique is partitioning your data and processing each partition individually (possibly on multiple computers). Depending on the problem you're trying to solve, there may be special data structures that you can use to reduce the amount of memory needed to represent the data - a good example is a sparse matrix.

It's very unusual to need this much memory - make sure to carefully consider if it's actually needed before you dwell into the extremely complex workarounds.