In Julia, I want to use addprocs
and pmap
inside a function that is defined inside a module. Here's a silly example:
module test
using Distributions
export g, f
function g(a, b)
a + rand(Normal(0, b))
end
function f(A, b)
close = false
if length(procs()) == 1 # If there are already extra workers,
addprocs() # use them, otherwise, create your own.
close = true
end
W = pmap(x -> g(x, b), A)
if close == true
rmprocs(workers()) # Remove the workers you created.
end
return W
end
end
test.f(randn(5), 1)
This returns a very long error
WARNING: Module test not defined on process 4
WARNING: Module test not defined on process 3
fatal error on fatal error on WARNING: Module test not defined on process 2
43: : WARNING: Module test not defined on process 5
fatal error on fatal error on 5: 2: ERROR: UndefVarError: test not defined
in deserialize at serialize.jl:504
in handle_deserialize at serialize.jl:477
in deserialize at serialize.jl:696
...
in message_handler_loop at multi.jl:878
in process_tcp_streams at multi.jl:867
in anonymous at task.jl:63
Worker 3 terminated.
Worker 2 terminated.ERROR (unhandled task failure): EOFError: read end of file
WARNING: rmprocs: process 1 not removed
Worker 5 terminated.ERROR (unhandled task failure): EOFError: read end of file
4-element Array{Any,1}:Worker 4 terminated.ERROR (unhandled task failure): EOFError: read end of file
ERROR (unhandled task failure): EOFError: read end of file
ProcessExitedException()
ProcessExitedException()
ProcessExitedException()
ProcessExitedException()
What I'm trying to do is write a package that contains functions that perform operations that can be optionally parallelized at the user's discretion. So a function like f
might take an argument par::Bool
that does something like I've shown above if the user calls f
with par = true
and loops otherwise. So from within the definition of f
(and within the definition of the module test
), I want to create workers and broadcast the Distributions package and the function g
to them.
What's wrong with using @everywhere
in your function? The following, for example, works fine on my computer.
function f(A, b)
close = false
if length(procs()) == 1 # If there are already extra workers,
addprocs() # use them, otherwise, create your own.
@everywhere begin
using Distributions
function g(a, b)
a + rand(Normal(0, b))
end
end
close = true
end
W = pmap(x -> g(x, b), A)
if close == true
rmprocs(workers()) # Remove the workers you created.
end
return W
end
f(randn(5), 1)
Note: when I first ran this, I needed to recompile the Distributions
package since it had been updated since I had last used it. When I first tried the above script right after the recompiling, it failed. But, I then quit Julia and reopened it and it worked fine. Perhaps that was what is causing your error?
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With