It appears I've run out of 32-bit address space. What are my options?

Question

I'm trying to take the covariance of a large matrix using numpy.cov. I get the following error:

Python(22498,0xa02e3720) malloc: *** mmap(size=1340379136) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug

Process Python bus error

It seems that this is not uncommon for 32-bit machines/builds (I have a 64-bit mac os x 10.5, but using a 32-bit python and numpy build as I had trouble building numpy+scipy+matplotlib on a 64-bit installation).

So at this point what would be the recommended course of action that will allow me to proceed with the analysis, if not switching machines (none others are available to me at the moment)? Export to fortran/C? Is there a simple(r) solution? Thanks for your suggestions.

Answer 1

To be at your place, I would try to "pickle" (save) the matrix on my hard drive, close python , then in command line re-open the pickeled file and do my computation on a "fresh python" instance.

I would do that because maybe your problem is before computing the covariance.

import cPickle
import numpy
M = numpy.array([[1,2],[3,4]]) # here it will be your matrix
cPickle( M , open( "~/M.pic", "w") ) # here it's where you pickle the file

Here you close python. Your file should be saved in you home directory as "M.pic".

import cPickle
import numpy
M = cPickle.load( open( "~/M.pic", "r") )
M = numpy.coa( M )

If it still does not work, try setting a "good" dtype for your data. numpy seams to use dtype 'float64' of 'int64' by default. This is huge and if you do not need this precision, you might want to reduce it to 'int32' or 'float32'.

import numpy
M = numpy.array([[1,2],[3,4]] , dtype.float32 )

Indeed, I can guarantee you that C/Fortran is not an option for you. Numpy is already written in C/Fortran and probably by people cleverer than you and me ;)

By curiosity, how big is your matrix? how big is your pickled file?