How do you get the mean and std of a column in a csr_matrix?

Question

I have a sparse 988x1 vector (a column in a csr_matrix) created through scipy.sparse. Is there a way to gets its mean and standard deviation without having to convert the sparse matrix to a dense one?

numpy.mean seems to only work for dense vectors.

Answer 1

Since you are performing column slicing, it may be better to store the matrix using CSC rather than CSR. But that would depend on what else you are doing with the matrix.

To calculate the mean of a column in a CSC matrix you can use the mean() function of the matrix.

To calculate the standard deviation efficiently is going to involve just a bit more effort. First of all, suppose you get your sparse column like this:

col = A.getcol(colindex)

Then calculate the variance like so:

N = col.shape[0]
sqr = col.copy() # take a copy of the col
sqr.data **= 2 # square the data, i.e. just the non-zero data
variance = sqr.sum()/N - col.mean()**2