I have a sparse matrix structured similar to this, but much larger.
library(Matrix)
dfmtest<-new("dgCMatrix"
, i = c(0L, 1L, 2L, 4L, 5L, 6L, 8L, 0L, 1L, 2L, 3L, 4L, 6L, 7L, 8L,
0L, 2L, 3L, 6L, 7L, 8L, 1L, 2L, 4L, 5L, 6L, 7L, 8L, 9L, 0L, 1L,
3L, 4L, 5L, 6L, 7L, 8L, 9L, 0L, 1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L,
9L, 0L, 1L, 3L, 4L, 6L, 7L, 8L, 9L, 0L, 2L, 3L, 5L, 6L, 7L, 9L,
0L, 1L, 2L, 3L, 4L, 5L, 6L, 8L, 9L, 0L, 1L, 2L, 3L, 4L, 5L, 6L,
7L, 9L)
, p = c(0L, 7L, 15L, 21L, 29L, 38L, 48L, 56L, 63L, 72L, 81L)
, Dim = c(10L, 10L)
, Dimnames = list(NULL, NULL)
, x = c(4, 3, 1, 2, 3, 1, 2, 1, 3, 3, 2, 3, 3, 3, 4, 2, 1, 2, 3, 2,
1, 4, 1, 2, 2, 3, 2, 3, 4, 1, 4, 1, 3, 4, 3, 2, 2, 2, 4, 1, 2,
2, 1, 2, 3, 1, 1, 1, 4, 1, 1, 2, 1, 1, 1, 4, 3, 3, 2, 1, 2, 2,
1, 1, 3, 3, 4, 1, 2, 4, 2, 4, 1, 2, 2, 3, 4, 2, 1, 2, 4)
, factors = list()
)
I would like to be able to find the mean of each column (and row eventually), excluding the 0 values. If I attempt to do it manually I run into memory issues because of the size of my sparse matrix.
nzmean <- function(x) {
mean(x[x!=0])
}
dfmmeans <- apply(dfmtest, 2, nzmean)
# 1 2 3 4 5 6 7 8
#2.285714 2.750000 1.833333 2.625000 2.444444 1.800000 1.875000 2.000000
# 9 10
#2.666667 2.333333
When I run the above on my actual matrix I get the following error:
Error in asMethod(object) :
Cholmod error 'problem too large' at file ../Core/cholmod_dense.c, line 105
I have also looked into using the colMeans
function, but it looks as though it is including all 0 values in the calculation.
dfmmeans <- colMeans(dfmtest)
#[1] 1.6 2.2 1.1 2.1 2.2 1.8 1.5 1.4 2.4 2.1
Is there a good way to do this on a large sparse matrix?
Matrix has a nice summary
method that returns an i, j, x data frame of the non-zero elements in the matrix, which can easily be summarized using aggregate
(or dplyr or data.table, if you like):
library(Matrix)
str(summary(dfmtest))
#> Classes 'sparseSummary' and 'data.frame': 81 obs. of 3 variables:
#> $ i: int 1 2 3 5 6 7 9 1 2 3 ...
#> $ j: int 1 1 1 1 1 1 1 2 2 2 ...
#> $ x: num 4 3 1 2 3 1 2 1 3 3 ...
#> - attr(*, "header")= chr "10 x 10 sparse Matrix of class \"dgCMatrix\", with 81 entries"
aggregate(x ~ j, summary(dfmtest), mean)
#> j x
#> 1 1 2.285714
#> 2 2 2.750000
#> 3 3 1.833333
#> 4 4 2.625000
#> 5 5 2.444444
#> 6 6 1.800000
#> 7 7 1.875000
#> 8 8 2.000000
#> 9 9 2.666667
#> 10 10 2.333333
If you'd like a purely matrix ops version, you can use abs(sign(...))
to turn all non-sparse elements into ones, which lets you calculate column means only with colSums
:
colSums(dfmtest) / colSums(abs(sign(dfmtest)))
#> [1] 2.285714 2.750000 1.833333 2.625000 2.444444 1.800000 1.875000
#> [8] 2.000000 2.666667 2.333333
It is true that colMeans
does not support removal of zeros:
getMethod("colMeans", "dgCMatrix")
#Method Definition:
#
#function (x, na.rm = FALSE, dims = 1, ...)
#{
# .local <- function (x, na.rm = FALSE, dims = 1, sparseResult = FALSE)
# .Call(dgCMatrix_colSums, x, na.rm, sparseResult, FALSE, TRUE)
# .local(x, na.rm, dims, ...)
#}
#<environment: namespace:Matrix>
so we need to work out our own function.
colMeans_drop0 <- function (dgCMat) {
nnz_per_col <- diff(dgCMat@p)
ColInd <- rep.int(1:ncol(dgCMat), nnz_per_col)
sapply(split(dgCMat@x, ColInd), mean)
}
colMeans_drop0(dfmtest)
# 1 2 3 4 5 6 7 8
#2.285714 2.750000 1.833333 2.625000 2.444444 1.800000 1.875000 2.000000
# 9 10
#2.666667 2.333333
Note: columns with all zeros are ignored. Similarly:
rowMeans_drop0 <- function (dgCMat) {
RowInd <- dgCMat@i + 1
sapply(split(dgCMat@x, RowInd), mean)
}
and rows with all zeros are ignored.
Remarks
alistaire's answer is also good.
The summary
+ aggregate
approach is a different implementation of the idea in this answer.
getMethod("summary", "sparseMatrix")
#Method Definition:
#
#function (object, ...)
#{
# d <- dim(object)
# T <- as(object, "TsparseMatrix")
# r <- if (is(object, "nsparseMatrix"))
# data.frame(i = T@i + 1L, j = T@j + 1L)
# else data.frame(i = T@i + 1L, j = T@j + 1L, x = T@x)
# attr(r, "header") <- sprintf("%d x %d sparse Matrix of class \"%s\", with %d entries",
# d[1], d[2], class(object), length(T@i))
# class(r) <- c("sparseSummary", class(r))
# r
#}
#<environment: namespace:Matrix>
summary
first coerces any sparse matrix class to "dgTMatrix"
class, i.e., the triplet format, and the aggregate
relies on split
+ lapply
internally.
The idea of using colSums
can be desirable, if you want to retain the result (which is 0 of course) for all-zero columns.
Discussion with 20650
An colSums
/ rowSums
based implementation for our functions is also possible.
colMeans_drop0 <- function (dgCMat) {
nnz_per_col <- diff(dgCMat@p)
nnz_per_col[nnz_per_col == 0] <- 1 ## just avoid doing 0 / 0
setNames(colSums(dgCMat) / nnz_per_col, 1:ncol(dgCMat))
}
rowMeans_drop0 <- function (dgCMat) {
RowInd <- dgCMat@i + 1
nnz_per_row <- tabulate(RowInd)
nnz_per_row[nnz_per_row == 0] <- 1 ## just avoid doing 0 / 0
setNames(rowSums(dgCMat) / nnz_per_row, 1:nrow(dgCMat))
}
Since colSums
/ rowSums
drops dimnames, we add them in with setNames
. These two functions retain results for all-zero columns / rows. We also use tabulate
function to compute number of non-zero entries on rows efficiently.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With