Is there a non-template metalanguage for c++?

Question

I'm reading the c++ template meta-programming book from the boost::MPL authors. I'am a big fan of the spirit/phoenix libraries.

However I was wondering. Using templates for c++ meta-programming is a hassle. It was not intended for that use. Neither templates nor macros support looping. In templates you basically have to go through recursion.

As I understand it, the c++ compilation process is roughly:

1. expands/evaluates Macro
2. instantiates templates ( code generation )
3. then actual other compilation steps ( lexical analysis, parsing, intermediate code generation, machine independent optimization, generating assembly/object files/linking...)

template meta-programming is a hack into the steps 1 & 2, discovered somewhat by mistake 20 years ago.

wouldn't it be simpler to use an actual language ? I mean some language which could access the c++ syntax tree and manipulate it, together with nice text processing capability.

I am thinking of a template language like twig/django, together with appropriate hooks into the g++ AST, and native definitions for type_traits and pod types.

Does this exist ? I haven't found anything like this on the web. It could achieve the same result ( and more ) than c++ template meta programming, but with faster compilation, cleaner and more readable code.

Answer 1

What you want is a program transformation system (PTS).

These tools generalize metaprogramming by stepping outside the programming language entirely (so, unlike templates [and your complaint] PTSs are not limited to what you can say in the programming language).

In general, PTS are parameterized by programming language syntax and prettyprinting rules (enabling it to parse the language of interest P into ASTs, and to unparse from ASTs back to source code), a set of transformations (which represent the essence of the changes you want to make, e.g, "optimize array accesses") and some "metaprogram" M which sequences how the transformations are applied to acheive the exact purpose you want (e.g., "optimize these array accesses but not those)".

A good PTS allows you to write the transformations in a pattern language embedded in the syntax of P. A transformation thus states essentially, "If you match a pattern p, replace it by q" where p and q are surface syntax versions of the code of interest. For AST-based transformation rules, no one transformation can do arbitrarily large amounts of work (because pattern ASTs are fixed-diameter), so one needs more than one transformation to achieve complex results.

The metaprogram sequences the transformations; it may be coded in a variety of ways. Stratego codes meta programs in a "strategies" DSL extension to the base transform pattern language, which are reactions to transforms succeeding or failing at point, and tree navigation steps (up, down, repeat). TXL's meta programs are functional over pattern matches. DMS uses PARLANSE, a parallel programming language, which allows transforms to be applied (carefully) in parallel on really big sets of source code. Clang doesn't have full source pattern-matching, but has some "AST" matchers to make syntax matching a little easier; otherwise you write the transforms as procedural code walking up/down the tree, embedded in a meta program coded in C++ itself. Rose Compiler's meta programs are largely procedural like Clang. There the Eclipse CDT Refactoring tools; I don't know their status well, but they do not have pattern directed transformations, and I think the meta programs are written in Java. There are other PTSs; most of them are not as robust or mature as these. GCC exists but isn't organized to be a PTS. (Take all this with a grain of salt; I'm the guy behind DMS and so you might not want to believe me).

You specifically want to manipulate C++. This means that the tool must process C++ decently. Stratego and TXL do not presently support C++, and are unlikely because of the effort it takes to produce a full C++ front end. Clang obviously has a useful C++ front end. So does Rose by virtue of its use of the EDG front end. DMS has a full C++11 front end that we built and validated against on large GCC and Windows c++ code bases. I'm not sure about the maturity of the CDT front end; in particular, I don't understand what they are using as a parser.

(EDIT Sept 2015: DMS now handles full C++14 in both MS and GCC dialects).

What you trade by metaprogramming from "inside" vs. "outside" the language is readability of truly, truly arcane "template metaprogramming" for the usually less arcane syntax of patterns and the metaprogramming language. IMHO you get expressive power, because the PTSs are designed to manipulate source code, and they can typically call on analyzers provided by the underlying machinery (symbol tables, type information, control/dataflows) that the templates simply cannot get near.

For me, the real strength is that PTSs they can apply transformations across arbitrary code boundaries; most C++ templates can only generate code to replace the template call. (In particular, a flaw with all metatprogramming/reflection schemes implemented inside a language is they are invariably limited in capability; PTSs have no such limitations).

At my bio, you can find a link to a description of DMS. More importantly, there's a set of papers there (e.g., "Case Study: Re-engineering C++ Component Models Via Automatic Program Transformation"), describing massive transformation applied to C++ programs (to shift them from one RTOS with complex APIs to a completely different system, as well as refactor the APIs radically) that can give you a flavor for what this looks like. (To my knowledge, of the various PTS tools, only DMS has been applied to C++ transformation tasks this ambitious.) These papers will show that PTS operations don't resemble template metaprogramming at all, don't rely on tricks like SFINAE, and can do things no template metaprogram IMHO could ever achieve.