scipy large sparse matrix

Question

I'm trying to use large 10^5x10^5 sparse matrices but seem to be running up against scipy:

n = 10 ** 5
x = scipy.sparse.rand(n, n, .001)

gets

ValueError: Trying to generate a random sparse matrix such as the
    product of dimensions is greater than 2147483647 - this is not
    supported on this machine

Does anyone know why limit is there and if I can avoid it? (fyi, I'm using a macbook air with 4gb memory and the enthought distribution)

Answer 1

This is a limitation which results from the way scipy.sparse.rand() is implemented. You can roll your own random matrix generation to circumvent this limitation:

n = 10 ** 5
density = 1e-3
ij = numpy.random.randint(n, size=(2, n * n * density))
data = numpy.random.rand(n * n * density)
matrix = scipy.sparse.coo.coo_matrix((data, ij), (n, n))