Increasing mc.cores beyond the number of logical cores

Question

Playing around with R function parallel::mclapply, I found that argument mc.cores can be chosen greater than the number of logical cores (as indicated by parallel::detectCores), resulting in speedup greater than the number of logical cores. Here's a minimal example (for me, this worked on MacOS and Linux):

sleepy <- function(i) {
    start <- Sys.time()
    Sys.sleep(i)
    as.numeric(Sys.time() - start)
}

mc.cores <- 100L
ntasks   <- 10000L

start <- Sys.time()
out <- parallel::mclapply(2/ntasks*runif(ntasks), sleepy, mc.cores = mc.cores)

real_duration <- as.numeric(Sys.time() - start)
cpu_duration <- sum(unlist(out))

data.frame(logical.cores = parallel::detectCores(),
           mc.cores      = mc.cores,
           speedup       = cpu_duration/real_duration)


##   logical.cores mc.cores  speedup
## 1             8      100 30.49574

I also tried this out in an more realistic example, i.e. close to the real scenario I want to parallelize: this didn't result in any problem, either.

In the documentation of / tutorials on parallel::mclapply, I could not find any example where mc.cores > detectCores() is chosen, and most probably, there's a very good reason for it.

Could somebody explain what are the problems with this practice? Can it be reasonable in certain circumstances, e.g. when memory requirements are not an issue?

Answer 1

I sometimes use mc.cores > detectCores() to throttle memory usage. If you split a job into 10 parts and process them with mclapply and mc.preschedule=F, each core will only process 10% of your job at a time. If mc.cores was set to two, for example, the other 8 "nodes" would have to wait until one part finished before starting a new one. This can be desirable if you're running into memory issues and want to prevent each loop from taking on more than it can handle.