distributed.worker Memory use is high but worker has no data to store to disk

Question

distributed.worker - WARNING - Memory use is high but worker has no data to store to disk.  Perhaps some other process is leaking memory?  Process memory: 3.91 GB -- Worker memory limit: 2.00 GB
distributed.worker - WARNING - Worker is at 41% memory usage. Resuming worker. Process memory: 825.12 MB -- Worker memory limit: 2.00 GB

The above error appears when I try to run a piece of code that applies an algorithm to a dataset that I have. Having read through the documentation at https://distributed.dask.org/en/latest/worker.html, it's still not clear to me what the impact of this error will be on the results of this application. Does this just affect the speed or efficiency of this code, or will it impact my results?

Answer 1

That warning is saying that your process is taking up much more memory than you are saying is OK. In this situation Dask may pause execution or even start restarting your workers.

The warning also says that Dask itself isn't holding on to any data, so there isn't much that it can do to help the situation (like remove its data). My guess is that some of the libraries that you are using taking up a lot of memory. You might want to use Dask workers that have more than 2GB of memory.