Julia - Modules and Parallelism

Question

I'm having trouble understanding how to use worker processes along with modules. I'll try to explain my difficulty simply.

I have on my main process (process 1) a module, module A.

On worker processes ,I have module B, which is supposed to handle numerous complicated calculations (as is often the case with parallel workers).

The problem is that it seems as if module A needs to be defined on the workers...

The problem is that module A is much larger than module B, it includes thousands of lines of code, but module B only uses around 15 short functions.

Is there any workaround so I can make my main worker have access to module A but the workers access to module B but be able to call workers from methods in module A, and make them run functions defined in their module B? For example, there could be a method in module B called calculate_stuff().

The structure I was hoping to achieve is something like:

module A # main worker process using this module 
    function call_worker_and_calculate()
        remotecall_fetch(calculate_stuff, 2)
    end 

    export call_worker_and_calculate
end

module B # worker process 2 is using this module  
    function calculate_stuff()
        # some stuff 
    end 

    export calculate_stuff 
end 

This particular example would return the error message:

julia> A.call_worker_and_calculate()
ERROR: On worker 2:
UndefVarError: A not defined

Answer 1

The error message is odd (I can't reproduce it with your code, what version do you use?). So I'm not sure this answers your question.

If you want to use a name from a module B in another module A, you must import or using module B in A.

For this to work, B must be in a path contained in the Julia variable LOAD_PATH or defined on all workers with @everywhere before the definition of A, e.g.,

@everywhere module B  # defined on all workers
    function calculate_stuff()
        # do stuff 
    end 

    export calculate_stuff 
end

module A # only defined on the main worker process
    using B  # introduces calculate_stuff into A's scope because it is exported in B
    function call_worker_and_calculate()
        remotecall_fetch(calculate_stuff, 2)    
    end 

    export call_worker_and_calculate
end

Then A.call_worker_and_calculate() works and A is also not defined on other workers:

julia> remotecall_fetch(whos,2)
    From worker 2:                               B   4537 bytes  Module
    From worker 2:                            Base  34048 KB     Module
    From worker 2:                            Core  12482 KB     Module
    From worker 2:                            Main  40807 KB     Module