python matrix multiplication: how to handle very large matrices?

Question

a = numpy.zeros((17770,5))

b = numpy.zeros((5,20000))

ma = numpy.matrix(a)

mb = numpy.matrix(b)

That is, ma.shape = (17770,5), mb.shape = (5,20000), both are numpy.matrix.

I need ma*mb. But I get the error message "ValueError: array is too big".

Are these matrices too large for Python multiplication?

BTW, I tested with python2.6.6/32bit/3GB RAM

Answer 1

I can compute ma*mb on my machine (Python 2.7.1 |EPD 7.0-2 (64-bit) on 64-bit Ubuntu).

Make sure you're using 64-bit Python on a 64-bit OS since a 17770x20000 matrix of double-precision floats requires 2.8GB of RAM, which exceeds (or is very close to) what most 32-bit platforms can handle.

Depending on your requirements, using single-precision floats (numpy.float32) might also be a possibility.

Lastly, if your matrices are sparse or have structure you might want to look into exploiting that to reduce memory requirements.