Programmatically/Dynamically import modules in Julia

Question

I was wondering if there was a way to programmatically or dynamically import a set of modules into Julia? For example, if I have a list of files that conform to some naming convention that are present at startup that I can grab using something like:

module_files = filter(r"^mod[0-9][0-9].jl$", readdir())

which might return a list of files ["mod00.jl", "mod02.jl", "mod05.jl"], is there a way to then import each of the modules in those files. This would be equivalent to having:

import mod00
import mod02
import mod05

in the code if I knew that those modules were available when I wrote the code. Or maybe there is some other approach to doing this that is better. Any suggestions would be much appreciated.

Update

I attempted to do this via a macro, but with no luck. For example:

macro import_mod(modn)
    quote
        import $modn
    end
end

function test_mimport()
    module_files = filter(r"^mod[0-9][0-9].jl$", readdir())
    println(module_files)
    for modname in factor_files
        modn = modname[1:end-3]
        println(modn)
        @import_mod modn
    end
end

When I run this, I get ERROR: syntax: invalid "import" statement. I tried various escaping strategies but all failed similarly.

Answer 1

NOTE: Please refer to user1712368's (Jameson Nash, Julia dev) answer, this discussion at the julia-users mailing list and this entry of the Julia manual, to know about why this is not the correct answer.

This is how a multiple import expression looks like, using Julia version 0.3.4:

julia> parse("import Foo, Bar")
:($(Expr(:toplevel, :($(Expr(:import, :Foo))), :($(Expr(:import, :Bar))))))

julia> dump(ans)
Expr 
  head: Symbol toplevel
  args: Array(Any,(2,))
    1: Expr 
      head: Symbol import
      args: Array(Any,(1,))
        1: Symbol Foo
      typ: Any
    2: Expr 
      head: Symbol import
      args: Array(Any,(1,))
        1: Symbol Bar
      typ: Any
  typ: Any

Here is a macro which does this programmatically, it takes a modules argument that could be a :call or :vcat Expr or a Symbol, which has to evaluate to a Vector{Symbol}:

julia> macro dynamic_import(modules)
           (modules = eval(modules))::Vector{Symbol}
           ex = Expr(:toplevel)
           for m in modules
               push!(ex.args, Expr(:import, m))
           end
           return ex
       end

You could also generalize this line:

module_files = filter(r"^mod[0-9][0-9].jl$", readdir())

By abstracting it into a function that takes a Regex and a String directory path as arguments and returns a Vector{Symbol}:

julia> function needed_modules(rx::Regex, dir::String=".")
           module_files = filter(rx, readdir(dir))
           module_syms = map(m -> symbol(split(m, '.')[1]), module_files)
       end
needed_modules (generic function with 2 methods)

So you may use it like this:

julia> @dynamic_import [:Mod01, :Mod02]    # :vcat expression

julia> rx = r"^Mod[0-9][0-9].jl$";

julia> @dynamic_import needed_modules(rx)    # :call expression

julia> modules = needed_modules(rx)
2-element Array{Symbol,1}:
 :Mod01
 :Mod02

julia> @dynamic_import modules    # Symbol

Finally you could wrap it all into a module so you may use using DynamicImport:

Note: currently I get this when trying to run the same examples from a module:

julia> using DynamicImport

julia> @dynamic_import [:mod01, :mod02]

julia> rx = r"^mod[0-9][0-9].jl$";

julia> @dynamic_import needed_modules(rx)
ERROR: rx not defined

julia> modules = needed_modules(rx)
2-element Array{Symbol,1}:
 :mod01
 :mod02

julia> @dynamic_import modules
ERROR: modules not defined

But it works fine If I define the objects inside the REPL, I guess this is an issue involving hygiene, which is something I'm not experienced with, so I'll ask at the julia-users mailing list.

Answer 2

The function version of import X is require("X"), so you should be able to do:

function test_mimport()
    module_files = filter(r"^mod[0-9][0-9].jl$", readdir())
    println(module_files)
    for modname in factor_files
        modn = modname[1:end-3]
        println(modn)
        require(modn)
    end
end

Then, assuming each of these defined a module of the same name, you can collect them into an array:

modules = Module[]
...
if isdefined(Main, symbol(modn))
    push!(modules, getfield(Main, symbol(modn))
else
    warn("importing $modn did not defined a module")
end
...
return modules