How to write an LLVM backend for stack machine?

Question

When I'm trying to build an LLVM backend without defining RegisterClass instances in TableGen file, it throws the following error:

error:No 'RegisterClass' subclasses defined

How to define a stack machine target for LLVM (i.e. that doesn't use registers)?

Answer 1

Just don't do it. LLVM DAG is designed specifically for the register machines and is of a little value for a stack machine compilation. You'd need a custom backend instead.

One approach I was using for a similar problem (LLVM IR -> stack soft core on an FPGA) is following:

1) Out of SSA (reg2mem)

2) Build an intermediate tree representation (i.e., squash all the single-use register assignments into trees, if order allows). This is the main optimisation bottleneck, the less register assignments you leave, the better.

3) "Register allocation" for the remaining registers that all are represented as stack allocated variables now (hope your stack machine ISA allows it).

4) Translate the tree directly, without any instruction selection (you can do it on LLVM IR level instead with intrinsics and a custom instcombine pass).

EDIT: It's not easy to do this with the DAG based backend, it was never meant to be used this way.

The closest thing I can think about is to use the DAG backend to generate an intermediate register machine code, and then use a post-processing pass to re-order the instructions where possible and inject the stack allocated variables access otherwise. I.e., represent each of the stack machine instructions as a pseudo-instruction with 1 or 2 register arguments and 1 register result.