How to parse multiline string in Julia?

Question

How can I parse more lines of code?

This is working:

julia> eval(parse("""print("O");print("K")"""))
OK

This is not working:

julia> eval(parse("""print("N");
print("O")"""))
ERROR: ParseError("extra token after end of expression")
Stacktrace:
 [1] #parse#235(::Bool, ::Function, ::String) at ./parse.jl:237
 [2] parse(::String) at ./parse.jl:232

BTW if I try line by line I have other problems. For example:

julia> parse("""for i in 1:3""")
:($(Expr(:incomplete, "incomplete: premature end of input")))

although:

julia> eval(parse("""for i in 1:2
println(i)
end"""))
1
2

Answer 1

It's a bit dopey, but this will do the trick!:

function parseall(str)
    return Meta.parse("begin $str end").args
end

---

Explanation:

As @Alexander Morley pointed out above, parse is designed to only parse a single expression. So if you just make that single expression able to contain all the expressions in your original string, then it will parse it just fine! :)

The simplest way to do that is to wrap the string in a block, which makes it a valid julia expression. You can see that such an expression parses just fine:

julia> Meta.parse("""
       begin
       function f3(x)
         x + 2
       end
       print("N")
       print(f3(5))
       end
       """)
>> quote
    #= none:2 =#
    function f3(x)
        #= none:3 =#
        x + 2
    end
    #= none:5 =#
    print("N")
    #= none:6 =#
    print(f3(5))
end

This is in fact the structure that the code in @Alexander Morley's is building. (And that's what made me think of doing it this way! Thanks Alexander!)

However, do note that if you are trying to parse a file, you actually don't want to return a single block expression like that returns, because you can't eval it as a single block: Julia doesn't allow you to nest some "top-level statements" (Modules) inside of anything. Instead you want to return an array of top-level expressions. That's why we return the args out of the Block expression.

With the above, you can eval each of those top-level expressions, and it will work correctly! :)

julia> for expr in parseall("module M  f() = 1  end");  Core.eval(Main, expr);  end

julia> M.f()
1

This part was figured out together with @misakawa (@thautwarm on Github).

Answer 2

parse is designed to parse a single expression (at least that's what the docs say: given this I'm actually a bit surprised your first example works , without throwing an error...).

If you want to parse mutliple expressions then you can take advantage of the fact that:

1. parse can take a second argument start that tells it where to start parsing from.
2. If you provide this start argument then it returns a tuple containing the expression, and where the expression finished.

to define a parseall function yourself. There used to be one in base but I'm not sure there is anymore. Edit: there is still in the tests see below

# modified from the julia source ./test/parse.jl
function parseall(str)
    pos = start(str)
    exs = []
    while !done(str, pos)
        ex, pos = parse(str, pos) # returns next starting point as well as expr
        ex.head == :toplevel ? append!(exs, ex.args) : push!(exs, ex) #see comments for info
    end
    if length(exs) == 0
        throw(ParseError("end of input"))
    elseif length(exs) == 1
        return exs[1]
    else
        return Expr(:block, exs...) # convert the array of expressions
                                    # back to a single expression
    end
end