Generate source code from AST with Antlr4 and StringTemplates

Question

If I have an AST and modify it, can I use StringTemplates to generate the source code for the modified AST?

I have successfully implemented my grammar for Antlr4. It generates the AST of a source code and I use the Visitor Class to perform the desired actions. I then modify something in the AST and I would like to generate the source code for that modified AST. (I believe it is called pretty-printing?).

Does Antlr's built in StringTemplates have all the functionality to do this? Where should one start (practical advice is very welcome)?

Answer 1

You can walk the tree and use string templates (or even plain out string prints) to spit out text equivalents that to some extent reproduce the source text.

But you will find reproducing the source text in a realistic way harder to do than this suggests. If you want back code that the original programmer will not reject, you need to:

• Preserve comments. I don't think ANTLR ASTs do this.
• Generate layout that preserves the original indentation.
• Preserve the radix, leading-zero count, and other "format" properties of literal values
• Renerate strings with reasonable escapes

Doing all of this well is tricky. See my SO answer How to compile an AST back to source code for more details. (Weirdly, the ANTLR guy suggests not using an AST at all; I'm guessing this is because string templates only work on ANTLR parse trees whose structure ANTLR understands, vs. ASTs which are whatever you home-rolled.)

If you get all of this right, what you are likely to discover is that modifying the parse tree/AST is harder than it looks. For almost any interesting task on complex languages, you need information which is not trivial to extract from the tree (e.g., what is the meaning of this identifier?, where is this variable used?,...) I call this the problem of Life After Parsing. My main point is that it takes a lot of machinery to modify ASTs and regenerate code; be aware of the size of your project.