Benefits of accessing the Abstract Syntaxt Tree (AST) . How does Julia exploit it?

Question

I have read that Julia has access to the AST of the code it runs. What exactly does this mean? Is it that the runtime can access it, that code itself can access it, or both?

Building on this:

1. Is this a key difference of Julia with respect to other dynamic languages, specifically Python?
2. What are the practical benefits of being able to access the AST?

What would be a good example of something that you can't easily do in Python, but that you can do in Julia, because of this?

Answer 1

What distinguishes Julia from languages like Python is that Julia allows you to intercept code before it is evaluated. Macros are just functions, written in Julia, which let you access that code and manipulate it before it runs. Furthermore, rather than treating code as a string (like "f(x)"), it's provided as a Julian object (like Expr(:call, :f, :x)).

There are plenty of things this allows which just aren't possible in Python. The main ones are:

You can do more work at compile time, increasing performance

Two good examples of this are regexes and printf. Both of these take a format specification of some kind and interpret it in some way. Now, these can fairly straightforwardly be implemented as functions, which might look like this:

match(Regex(".*"), str)
printf("%d", num)

The problem with this is that these specifications must be re-interpreted every time the statement is run. Every time the interpreter goes over this block, the regex must be re-compiled into a state machine, and the format must be run through a mini-interpreter. On the other hand, if we implement these as macros:

match(r".*", str)
@printf("%d", num)

Then the r and @printf macros will intercept the code at compile time, and run their respective interpreters then. The regex turns into a fast state machine, and the @printf statement turns into a simple println(num). At run time the minimum of work is done, so the code is blazing fast. Now, other languages are able to provide fast regexes, for example, by providing special syntax for it – but the fact that they're not special-cased in Julia means that developers can use the same techniques in their own code.

You can make mini-compilers for, well, pretty much anything

Languages with macros tend to have more capable embedded DSLs, because you can change the semantics of the language at will. For example, the algebraic modelling language, JuMP.jl. Clojure also has some neat examples of this too, like its embedded logic programming language. Mathematica.jl even embeds Mathematica's semantics in Julia, so that you can write really natural symbolic expressions like @Integrate(log(x), {x,0,2}). You can fake this to a point in Python (SymPy does a good job), but not as cleanly or as efficiently.

If that doesn't convince you, consider that someone managed to implement an interactive Julia debugger in pure Julia using macros. Try that in Python.

Edit: Another great example of something that's difficult in other languages is Cartestian.jl, which lets you write generic algorithms across arrays of any number of dimensions.