Specifying which objects to load in doSNOW clusters

Question

I have a basic question regarding doSNOW clusters in R. When we register parallel clusters in R, it loads all the variables (data) assigned in global environment to parallel cluster. Is there any way, we can specify which objects to load in parallel cluster?

To be more specific. Consider the following example. I have two variables x and y in global environment. When I register cluster it loads both x and y to the registered clusters. However, I need only x in parallel clusters and not y.

library(doSNOW)

cl <- makeCluster(2, type="SOCK")

registerDoSNOW(cl)


x <- 1:5

y <- 6:10

foreach(x=x, .combine=c) %dopar% x^2

stopCluster(cl)

As there is a time involved in the setting up the parallel cluster, it can help to reduce that time, when there are lots of data (or big datasets) in the global environment but you don't need all of them in parallel clusters.

Answer 1

Variables aren't exported to the workers when doSNOW is registered. Variables are only auto-exported to the workers when the foreach loop is executed, and even then it only exports variables that are actually referenced in the body of the foreach loop and defined in the "local" environment (which isn't the global environment if the foreach loop is executed in a function, for example). It uses this conservative approach to avoid sending huge objects to the workers that are not required for the computation.

It's also important to note that auto-exported variables are not exported to the global environment of the workers, even if they are defined in the global environment of the master. They are defined in a special environment setup by doSNOW so that they won't corrupt any global variables that you may have created using the clusterExport function, for example.

You can use the foreach .verbose=TRUE option to get a listing of the variables that are auto-exported by doSNOW. Here is the report for your example:

> foreach(x=x, .combine='c', .verbose=TRUE) %dopar% {
+   x ^ 2
+ }
no variables are automatically exported

If you want to prevent a particular variable from being auto-exported, you can use the foreach .noexport option. That might be useful if you wanted to export a large data frame once to the workers using clusterExport and then use it in multiple foreach loops.