Could not find function inside clusterApply

Question

I have this reproducible R snippet

rm(list=ls())

library(doSNOW)

f <- function(a, b) a+b
g <- function(c) f(c*c, c+c)

v <- c(1, 2, 3, 4, 5, 6)

cl <- makeMPIcluster(1)
cat( clusterApply(cl, v, g) )
stopCluster(cl)

and I get the following error message:

Error in checkForRemoteErrors(val) : 
  6 nodes produced errors; first error: could not find function "f"

I am using R 2.14.1 under Ubuntu. MPI is installed and working.

I know that there is a similar issue for the foreach construct, but it allows to reference functions manually through the .export parameter. I could not find anything similar for clusterApply. Is there a workaround for this?

Thanks!

Answer 1

Your function wasn't sent to the workers. Perhaps the best way to do this is to export the function directly:

clusterExport(cl, list("f", "g"))

Answer 2

I think your problem relate to "variable scope". On Mac/Linux you have the option of using makeCluster(no_core, type="FORK") that automatically contains all environment variables. On Windows you have to use the Parallel Socket Cluster (PSOCK) that starts out with only the base packages loaded. Thus, you always specifiy exactly what variables as well as library that you include for parallel function to work. clusterExport() and clusterEvalQ() are necessary so as to the function to see the needed variables and packages respectively. Note that any changes to the variable after clusterExport are ignored. Comeback to your problem. You must use as following:

rm(list=ls())
library(doSNOW)
f <- function(a, b) a+b
g <- function(c) f(c*c, c+c)
v <- c(1, 2, 3, 4, 5, 6)
cl <- makeMPIcluster(1)
# insert code here
clusterExport(cl, list("f", "g"))
# insert clusterEvalQ(cl, library(...)) if you need library for function to parallel
cat( clusterApply(cl, v, g) )
stopCluster(cl)