Are there any non-Lisp dialects that allow for syntactic abstraction?

Question

As Rich Hickey says, the secret sauce of Lisp languages is the ability to directly manipulate the Abstract Syntax Tree through macros. Can this be achieved in any non-Lisp dialect languages?

Answer 1

For a sake of completeness, in addition to the already mentioned languages and preprocessors:

• Template Haskell
• Katahdin
• MetaOCaml
• MetaLua (unless you account it as a Lisp dialect)
• Nemerle
• Converge

Answer 2

Being able to "directly manipulate the abstract syntax tree" by itself is nothing new, though it's something that very few languages have. For example, many languages these days have some kind of an eval function -- but it should be obvious that that's not manipulating the abstract syntax tree, instead, it is a manipulation of the concrete syntax -- the direct source code. Incidentally, the mentioned functionality in D falls under the same category, as is CPP: both deal with raw source text.

To give an example of a language that does have that feature (but not something that would be considered macros proper), see OCaml. It has a syntactic extension system, CamlP4, which is essentially a compiler extension toolkit, and it revolves around the OCaml abstract syntax as its most important purpose. But this is still not what makes the corresponding feature in Lisps so great.

The important feature of Lisps is that the extensions that you get using macros are part of the language in the same way that any other syntactic form is. To put this differently, when you use something like if in a Lisp, there is no difference in functionality whether it's implemented as a macro or as a primitive form. (Actually there is a minor difference: in some cases it's important to know the set of primitive forms that don't expand further.) More specifically, a Lisp library can provide plain bindings and macros, which means that libraries can extend the language in a much more interesting way than the usual boring extensions you get in most languages, capable of adding only plain bindings (functions and values).

Now, viewed in this light, something like the D facility is very similar in nature. But the fact that it deals with raw text rather than ASTs limit its utility. If you look at the example on that page,

mixin(GenStruct!("Foo", "bar"));

you can see how this doesn't look like part of the language -- to make it more like Lisp, you'd use it in a natural way:

GenStruct(Foo, bar);

with no need for a mixin keyword that marks where a macro is used, no need for that !, and the identifiers being specified as identifiers rather than strings. Even better, the definition should be expressed more naturally, something like (inventing some bad syntax here):

template expression GenStruct(identifier Name, identifier M1) {
    return [[struct $Name$ { int $M1$; }; ]]
}

One important thing to note here is that since D is a statically typed language, ASTs have crept into this mental exercise in an explicit way -- as the identifier and expression types (I'm assuming here that template marks this as a macro definition, but it still needs a return type).

In Lisp, you're essentially getting something very close to this functionality, rather than the poor string solution. But you get even more -- Lisp intentionally puns over the basic list type, and unifies the ASTs with the runtime language in a very simple way: the AST is made of symbols and lists and other basic literals (numbers, strings, booleans), and those are all part of the runtime language. In fact, for those literals, Lisp takes another step forward, and uses the literals as their own syntax -- for example, the number 123 (a value that exists at runtime) is represented by a syntax which is also the number 123 (but now it's a value that exists at compile-time). The bottom line of this is that macro-related code in Lisp tends to be far easier to deal with than what other languages call "macro"s. Imagine, for example, making the D example code create N int fields in a struct (where N is a new input to the macro) -- that would require using some function to translate a string into a number.