How to force idle workers to take jobs in parallel R?

Question

I am new to posting here--I searched and couldn't find an answer to my question. I have run the following R parallelized code (from a blog on parallel computing in R) using the parallel package on two different machines and yet get very different process time results. The first machine is a Lenovo laptop with Windows 8, 8GB RAM, Intel i7, 2 cores/4 logical processors. The second machine is a Dell desktop, Windows 7, 16GB RAM, Intel i7, 4 cores/8 logical processors. The code sometimes runs much slower on the second machine. I believe the reason is that the second machine is not using the worker nodes to complete the task. When I use the function snow.time() from the snow package to check node usage, the first machine is using all available workers to complete the task. However, on the more powerful machine, it never uses the workers--the entire task is handled by the master. Why is the first machine using workers, but the second is not with the exact same code? And how do I 'force' the second machine to use the available workers so that the code is truly parallelized and the processing time is sped up? The answers to these would help me tremendously with other work I am doing. Thanks in advance. The graphs from the function snow.time() are below as well as the code I used:

runs <- 1e7
manyruns <- function(n) mean(unlist(lapply(X=1:(runs/4), FUN=onerun)))

library(parallel)
cores <- 4
cl <- makeCluster(cores)

# Send function to workers
tobeignored <- clusterEvalQ(cl, {
    onerun <- function(.){ # Function of no arguments
        doors <- 1:3
        prize.door <- sample(doors, size=1)
        choice <- sample(doors, size=1)
        if (choice==prize.door) return(0) else return(1) # Always switch
    }
    ; NULL
})

# Send runs to the workers
tobeignored <- clusterEvalQ(cl, {runs <- 1e7; NULL})
runtime <- snow.time(avg <- mean(unlist(clusterApply(cl=cl, x=rep(runs, 4), fun=manyruns))))
stopCluster(cl)

plot(runtime)

Answer 1

Try clusterApplyLB instead of clusterApply. The "LB" is for load balancing.

The non LB version divides the number of tasks between the nodes and sends them in a batch, but if one node finishes early then it sits idle waiting for the others.

The LB version sends one task to each node then watches the nodes and when a node finishes it sends another task to that node until all the tasks are assigned. This is more efficient if the time for each task varies widely, but is less efficient if all the tasks will take about the same amount of time.

Also check the versions of R and parallel. If I am remembering correctly the clusterApply function used to not do things in parallel on Windows machines (but I don't see that note any more, so that has likely been remedied in recent versions), so the difference could be different versions of the parallel package. The parLapply function did not have the same issue, so you could rewrite your code to use it instead and see if that makes a difference.