Julia pmap: How to effectively send helper functions to other workers?

Question

When using pmap function, an error would occur if the function is not defined on any of the worker processes. However, when the function calls other functions or using other functions inside another .jl file, to use @everywhere macro on every related functions is certainly not a good solution.

Is there a neat way to effectively send a function along with its helpers to all available workers?

Answer 1

I do not think there is a macro that can be used with a function definition to send the definitions of all its helper functions to all the worker processes.

However, there are better ways to send all the functions you need than putting an @everywhere before each of them.

You can put all these functions in a file and include it everywhere with @everywhere include("mynewfile.jl"). If your functions use other functions inside another .jl file, put that include statement for the other file in mynewfile.jl as well. If you are using modules from the other file, put the using or import statements inside mynewfile.jl

In a similar way, instead of a file, you can use @everywhere begin...end block. Put all these functions, using or import statements, includes etc. into a begin...end block and put an @everywhere before the begin. This is especially useful if you are working with IJulia notebooks.

julia> @everywhere begin
           g(x) = x^2
           f(x) = g(x)*2
       end

julia> pmap(f, 1:5)
5-element Array{Int64,1}:
  2
  8
 18
 32
 50

You can also create modules/packages and just use a single @eveywhere using MyNewModule. If you are using modules outside a package, you should also include the definitions/file of that module everywhere.

You might find it useful to read the relevant manual entry.