Foreach combine output - specific vectors

Question

I am just getting started with foreach & parallel and encountered some challenges with combining my outputs from each iteration.

Currently, in each iteration, I am outputting a set of variables that I would like to combine into a vector, so that I can obtain some summary stats for across all iterations. An example:

foreach (s = 1:num) %dopar%{
var1 = ...
var2 = ...
var3 = ...}

Ultimately I'd like 3 vectors containing each iteration's produced output, so: var1 = c(var1 from s=1, var1 from s=2, var1 from s=3, ...)

I looked into the .combine option and attempted to create a function that would append, but found no success.

Answer 1

The {} after foreach...%dopar% behaves like a function, that is, it will return a value whether you explicitly specify it or not. It is generally not specified but by specifying you can return whatever value you like

foreach(I=1:3) %dopar% { <complicated stuff> 
                         return(1) }

# [[1]]
# [1] 1

# [[2]]
# [1] 1

# [[3]]
# [1] 1

In your case, you want to return 3 vectors, unfortunately this is not possible

library(foreach)
library(doParallel)
cl <- makeCluster(detectCores())
registerDoParallel(cl)

foreach(I=1:8) %dopar% { v1 <- "ant"
                 v2 <- "pant"
                 return(v1, v2) }

Which results in the following error

Error in { : task 1 failed - "multi-argument returns are not permitted"

But you can make a list of your vectors, and return the value

foreach(I=1:8, .combine=rbind) %dopar% {  v1 <- "ant"
                                          v2 <- "pant"
                                          return(list(v1, v2)) }

         # [,1]  [,2]  
# result.1 "ant" "pant"
# result.2 "ant" "pant"
# result.3 "ant" "pant"
# result.4 "ant" "pant"
# result.5 "ant" "pant"
# result.6 "ant" "pant"
# result.7 "ant" "pant"
# result.8 "ant" "pant"

stopCluster(cl)

Answer 2

Solution using future:
doFuture is future's wrapper for foreach which I prefer instead of parallel.
Here we create data.frame res and you can access vector using res$vector1

library(doFuture)
registerDoFuture()
plan(multiprocess)

Nlength <- 1000
Nvector <- 3

res <- foreach(i = 1:Nvector, .combine = cbind) %dopar% {
    1:Nlength / pi * rnorm(1)
}
res <- data.frame(res)
colnames(res) <- paste0("vector", 1:Nvector)

dim(res)
[1] 1000    3