How to shut down an R parallel cluster without the cluster variable?

Question

With the parallel R package, I can run things in parallel like this:

library(parallel)
cl <- makeCluster(2) # Create a cluster with 2 workers
... # do some parallel stuff
stopCluster(cl)

However the cl variable referring to the cluster can get lost, for instance when running from a function that fails:

do.something <- function() {
    library(parallel)
    cl <- makeCluster(detectCores())
    parLapply(cl, 1:10, function(x) {
        stop("An error occured")
    })
    stopCluster(cl)
}
do.something()

here, stopCluster has not been executed. When this happens, I am left with the workers running, as shown by ps:

501 53300  9225   0  2:16PM ttys003    0:00.27 /opt/local/Library/Frameworks/R.framework/Resources/bin/exec/R
501 53390     1   0  2:19PM ttys003    0:00.16 /opt/local/Library/Frameworks/R.framework/Resources/bin/exec/R --slave --no-restore -e parallel:::.slaveRSOCK() --args MASTER=localhost PORT=11099 OUT=/dev/null TIMEOUT=2592000 XDR=TRUE
501 53399     1   0  2:19PM ttys003    0:00.16 /opt/local/Library/Frameworks/R.framework/Resources/bin/exec/R --slave --no-restore -e parallel:::.slaveRSOCK() --args MASTER=localhost PORT=11099 OUT=/dev/null TIMEOUT=2592000 XDR=TRUE
501 53408     1   0  2:19PM ttys003    0:00.16 /opt/local/Library/Frameworks/R.framework/Resources/bin/exec/R --slave --no-restore -e parallel:::.slaveRSOCK() --args MASTER=localhost PORT=11099 OUT=/dev/null TIMEOUT=2592000 XDR=TRUE
501 53417     1   0  2:19PM ttys003    0:00.16 /opt/local/Library/Frameworks/R.framework/Resources/bin/exec/R --slave --no-restore -e parallel:::.slaveRSOCK() --args MASTER=localhost PORT=11099 OUT=/dev/null TIMEOUT=2592000 XDR=TRUE

Of course I could manually kill the slaves one by one, or restart R. However at times it may not be practical, for instance if multiple instances of R are running their own pools. Is there a way to stop them from within R when cl has been lost? How do people normally handle this scenario?

Answer 1

There are a couple of mechanisms to have code always run, even if there is an error:

try

Wrap the error-prone section inside a try or tryCatch block. You can then examine the result to see if there was an error.

do.something <- function() {
    library(parallel)
    cl <- makeCluster(detectCores())
    result <- try({
        parLapply(cl, 1:10, function(x) {
            stop("An error occured")
        })
    })
    if(inherits(result, "try-error"))
        print("there was an error!")
    stopCluster(cl)
    result
}

on.exit

The code inside an on.exit call will always run when the function ends, whether cleanly or due to an error.

do.something <- function() {
    library(parallel)
    cl <- makeCluster(detectCores())
    on.exit(stopCluster(cl))
    parLapply(cl, 1:10, function(x) {
        stop("An error occured")
    })
}