clusterExport, environment and variable scoping

Question

I wrote a function in which I define variables and load objects. Here's a simplified version:

fn1 <- function(x) {
  load("data.RData") # a vector named "data"
  source("myFunctions.R")
  library(raster)
  library(rgdal)

  a <- 1
  b <- 2
  r1 <- raster(ncol = 10, nrow = 10)
  r1 <- init(r1, fun = runif)
  r2 <- r1 * 100
  names(r1) <- "raster1"
  names(r2) <- "raster2"
  m <- stack(r1, r2) # basically, a list of two rasters in which it is possible to access a raster by its name, like this: m[["raster1"]]

  c <- fn2(m)
}

Function "fn2" is can be found in "myFunctions.R" and is defined as:

fn2 <- function(x) {
  fn3 <- function(y) {
   x[[y]] * 100 * data
  }

  cl <- makeSOCKcluster(8)   
  clusterExport(cl, list("x"), envir = environment()) 
  clusterExport(cl, list("a", "b", "data")) 
  clusterEvalQ(cl, c(library(raster), library(rgdal), rasterOptions(maxmemory = a, chunksize = b))) 
  f <- parLapply(cl, names(x), fn3)  
  stopCluster(cl)
}

Now, when I run fn1, I get an error like this:

Error in get(name, envir = envir) : object 'a' not found

From what I understand from ?clusterExport, the default value for envir is .GlobalEnv, so I would assume that "a" and "b" would be accessible to fn2. However, it doesn't seem to be the case. How can I access the environment to which "a" and "b" belong?

So far, the only solution I have found is to pass "a" and "b" as arguments to fn2. Is there a way to use these two variables in fn2 without passing them as arguments?

Thanks a lot for your help.

Answer 1

You're getting the error when calling clusterExport(cl, list("a", "b", "data")) because clusterExport is trying to find the variables in .GlobalEnv, but fn1 isn't setting them in .GlobalEnv but in its own local environment.

An alternative is to pass the local environment of fn1 to fn2, and specify that environment to clusterExport. The call to fn2 would be:

c <- fn2(m, environment())

If the arguments to fn2 are function(x, env), then the call to clusterExport would be:

clusterExport(cl, list("a", "b", "data"), envir = env)

Since environments are passed by reference, there should be no performance problem doing this.