Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Parallel version of sapply

Do we have parallel version of sapply, as we have mclapply in parallel package which is a version of lapply.

like image 713
Anuj Avatar asked Jun 25 '15 12:06

Anuj


1 Answers

As @RHertel points out, the snow package has apply family options, but it does not count with an mcsapply function as it is asked. If you take a look a the implementation of sapply, it is simply a call to lapply with some more post processing:

sapply
#> function (X, FUN, ..., simplify = TRUE, USE.NAMES = TRUE) 
#> {
#>     FUN <- match.fun(FUN)
#>     answer <- lapply(X = X, FUN = FUN, ...)
#>     if (USE.NAMES && is.character(X) && is.null(names(answer))) 
#>         names(answer) <- X
#>     if (!isFALSE(simplify) && length(answer)) 
#>         simplify2array(answer, higher = (simplify == "array"))
#>     else answer
#> }
#> <bytecode: 0x559a53feef18>
#> <environment: namespace:base>

Created on 2019-11-22 by the reprex package (v0.3.0)

Since the mclapply function is equivalent to lapply, you could actually write your own replacing some parts of sapply, here is an example:

# An mc-version of the sapply function.
mcsapply <- function (X, FUN, ..., simplify = TRUE, USE.NAMES = TRUE) {
  FUN <- match.fun(FUN)
  answer <- parallel::mclapply(X = X, FUN = FUN, ...)
  if (USE.NAMES && is.character(X) && is.null(names(answer))) 
    names(answer) <- X
  if (!isFALSE(simplify) && length(answer)) 
    simplify2array(answer, higher = (simplify == "array"))
  else answer
}

# Testing it out
library(parallel)

ans0 <- mcsapply(1:20, function(i) rnorm(1e4), mc.cores = 4)
ans1 <- sapply(1:20, function(i) rnorm(1e4))

# Same structure
str(ans0)
#>  num [1:10000, 1:20] 0.1792 0.1581 -0.1293 -0.0324 0.1836 ...
str(ans1)
#>  num [1:10000, 1:20] 1.304 1.355 -1.387 1.07 0.582 ...

Created on 2019-11-22 by the reprex package (v0.3.0)

Finally, you can always create a Fork cluster and use parSapply instead.

like image 177
gvegayon Avatar answered Oct 22 '22 20:10

gvegayon