Dask - WARNING - Worker exceeded 95% memory budget

Question

I am getting error

Dask - WARNING - Worker exceeded 95% memory budget.

I am working on a local PC with 4 physical and 8 virtual cores, I have tried the following:

Per...

Managing worker memory on a dask localcluster

...and the documentation here...

https://distributed.readthedocs.io/en/latest/worker.html#memory-management

...I have tried editing .config\dask\distributed.yaml to uncomment the bottom five lines...

distributed:
  worker:
    # Fractions of worker memory at which we take action to avoid memory blowup
    # Set any of the lower three values to False to turn off the behavior entirely
    memory:
      target: 0.60  # target fraction to stay below
      spill: 0.70  # fraction at which we spill to disk
      pause: 0.80  # fraction at which we pause worker threads
      terminate: 0.95  # fraction at which we terminate the worker

I have also tried the following in my code:

from dask.distributed import Client, LocalCluster

    worker_kwargs = {
        'memory_limit': '1G',
        'memory_target_fraction': 0.6,
        'memory_spill_fraction': 0.7,
        'memory_pause_fraction': 0.8,
    #     'memory_terminate_fraction': 0.95,
    }

    cluster = LocalCluster(ip='0.0.0.0', n_workers=8, **worker_kwargs)
    client = Client(cluster, memory_limit='4GB')

...with and without the memory_limit argument to the Client() function.

Any ideas?

Answer 1

If you do not want dask to terminate the worker, you need to set terminate to False in your distributed.yaml file:

distributed:
  worker:
    # Fractions of worker memory at which we take action to avoid memory blowup
    # Set any of the lower three values to False to turn off the behavior entirely
    memory:
      target: 0.60  # target fraction to stay below
      spill: 0.70  # fraction at which we spill to disk
      pause: 0.80  # fraction at which we pause worker threads
      terminate: False  # fraction at which we terminate the worker

(You might also want to set pause to False.)

The file is typically located at ~/.config/dask/distributed.yaml:

Caveat: Do not forget to uncomment the distributed:, worker: and memory: line. Otherwise the change will have no effect.