Reload Dask worker containers automatically on code change

Question

I have the Dask code below that submits N workers, where each worker is implemented in a Docker container:

default_sums = client.map(process_asset_defaults, build_worker_args(req, numWorkers))
future_total_sum = client.submit(sum, default_sums)
total_defaults_sum = future_total_sum.result()

where process_asset_defaults is a method in a worker.

The problem is that in a development environment when I change the worker's code I need to restart all the containers manually for the change to take effect.

Is there a way to reload the worker with the new code without restarting the workers?

Note: the code resides in a Docker volume, I change it directly in the volume with Visual Studio Code.

Answer 1

If the code that changes is the content of the functions, then it might be possible to use autoreload:

import importlib
importlib.reload(process_asset_defaults) # if this is the function that needs updating

See blog or docs for some further details.