Menu
I want to start a local cluster from python with a specific number of workers, and then connect a client to it.
cluster = LocalCluster(n_workers=8, ip='127.0.0.1')
client = Client(cluster)
But before, I want to check if there is an existing local cluster, started for example by the dask-scheduler command. Is there a way to do this ?
879
The Dask SchedulerThe Scheduler acts as a middle layer between the client and the workers, instructing workers to execute the actual computations requested by the client. It also helps the workers coordinate with each other, deciding who should do which tasks.
When we create a Client object it registers itself as the default Dask scheduler. All . compute() methods will automatically start using the distributed system. We can stop this behavior by using the set_as_default=False keyword argument when starting the Client.
If we start Dask using processes — as in the following code — we get 8 workers, one for each core, with each worker allotted 2 GB of memory (16 GB total / 8 workers, this will vary depending on your laptop).
After spinning up a Dask cluster, you can use client. dashboard_link to get a link to your dashboard. If you're using the distributed scheduler for local computation, the dashboard will be served at localhost:8787. This dashboard shows the real-time status of your cluster, resources, and computations.
There is no standard convention to check if a scheduler exists on your machine. The best you can do is try with a short timeout. The default port is 8786
from dask.distributed import Client, TimeoutError
try:
client = Client('tcp://localhost:8786', timeout='2s')
except TimeoutError:
pass
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
