MemoryError when creating a very large numpy array [duplicate]

Question

I'm trying to create a very large numpy array of zeros and then copy values from another array into the large array of zeros. I am using Pycharm and I keep getting: MemoryError even when I try and only create the array. Here is how I've tried to create the array of zeros:

import numpy as np

last_array = np.zeros((211148,211148))

I've tried increasing the memory heap in Pycharm from 750m to 1024m as per this question: https://superuser.com/questions/919204/how-can-i-increase-the-memory-heap-in-pycharm, but that doesn't seem to help.

Let me know if you'd like any further clarification. Thanks!

Answer 1

Look into using the sparse array capabilities within scipy:
scipy.sparse documentation.

There are a set of examples and tutorials on the scipy.sparse library here:
Scipy lecture notes: Sparse Matrices in SciPy

This may help you solve your memory issues, as well as make everything run faster.

---

To create an empty sparse array with values in certain positions as you asked in your comment:

Is there any way to create an empty array with values in certain positions, such as: last_array[211147][9] but everywhere else would be empty?

from scipy.sparse import *
values = [42]
row_ind = [211147]
col_ind = [9] 
last_array = csc_matrix((values, (row_ind, col_ind)), shape=(211148,211148))

print(last_array[211147,9])