Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Option "cores" from package doParallel useless on Windows?

On a Linux computer, following doParallel's vignette, I use doParallel::registerDoParallel() and then I use options(cores = N) where N is the number of cores I want to use with foreach.

I can verify with foreach::getDoParWorkers() that when I change the option cores, it automatically changes the number of cores used by foreach.

Yet, on Windows 10 (latest versions of R and packages), this option doesn't seem to have any effect as changing its value doesn't change the value of foreach::getDoParWorkers() (which is initialized at 3 when calling doParallel::registerDoParallel()).

Reproducible example:

doParallel::registerDoParallel()
options(cores = 1)
foreach::getDoParWorkers()
options(cores = 2)
foreach::getDoParWorkers()
options(cores = 4)
foreach::getDoParWorkers()

Is it a bug? Won't it work on Windows?

Edit: I know how to register parallel backends differently. The goal is to use doParallel::registerDoParallel() registering once (at the loading of my package) and then use an option to change the number of cores used. This is why I want it to work also on Windows.

like image 329
F. Privé Avatar asked Dec 23 '22 14:12

F. Privé


1 Answers

The answer from the maintainer of package doParallel, Rich Calaway:

Windows does not support forking, which is what the parallel (and doParallel) packages use the “cores” argument for. So, on Windows, all “cores” arguments are set to 1. To use multiple cores on Windows with doParallel, use makeCluster to create a multiple worker cluster cl, then registerDoParallel(cl).

So this isn't a bug, but a non-Windows feature, which is a pity.

like image 52
F. Privé Avatar answered Jan 12 '23 21:01

F. Privé