Is it possible to shutdown a dask.distributed cluster given a Client instance?

Question

if I have a distributed.Client instance can I use that to shutdown the remote cluster? i.e. to kill all workers and also shutdown the scheduler?

If that can't be done using the Client instance is there another way other than manually killing each remote process?

Answer 1

There is no client function specifically for this.

The scheduler has a close() method which you could call using run_on_scheduler thus

c.run_on_scheduler(lambda dask_scheduler=None: 
    dask_scheduler.close() & sys.exit(0))

which will tell workers to disconnect and shutdown, and will close all connections before terminating the process. Note that this raises an error in the client, since the connection is broken without a reply. There are probably more elegant ways.

Note that the right way to do this is probably to interact with one of the deployment cluster managers. For example, LocalCluster has a user-facing close() method that you can call directly.

--EDIT--

client.shutdown() is now available.