Dask: set multiprocessing method from Python

Question

Is there a way to set the multiprocessing method from Python? I do not see a method in the Client() API docs of Dask.distributed that indicates how to set this property.

Update:

For example, is there:

client = Client(multiprocessing='fork')

or

client = Client(multiprocessing='spawn')

?

Answer 1

Unfortunately the multiprocessing context method is set at import time of dask.distributed. If you wanted to set this from Python you could set the config value after you import dask, but before you import dask.distributed.

import dask
dask.config.set({'distributed.worker.multiprocessing-method': 'spawn'})

from dask.distributed import Client

However it's probably more robust to just set this in your config file. See configuration documentation for the various ways to set configuration values.

Note: this is using the configuration as of dask.__version__ == '0.18.0'