How to write a X86_64 _assembler_?

Question

Goal: I want to write an X86_64 assembler. Note: marked as community wiki

Background: I'm familiar with C. I've written MIPS assembly before. I've written some x86 assembly. However, I want to write an x86_64 assembler -- it should output machine code that I can jump to and start executing (like in a JIT).

Question is: what is the best way to approach this? I realize this problem looks kind large to tackle. I want to start out with a basic minimum set:

• Load into register
• Arithmetric ops on registers (just integers is fine, no need to mess with FPU yet)
• Conditionals
• Jumps

Just a basic set to make it Turing complete. Anyone done this? Suggestions / resources?

Answer 1

An assembler, like any other "compiler", is best written as a lexical analyser feeding into a language grammar processor.

Assembly language is usually easier than the regular compiled languages since you don't need to worry about constructs crossing line boundaries and the format is usually fixed.

I wrote an assembler for a (fictional) CPU some two years ago for educational purposes and it basically treated each line as:

• optional label (e.g., :loop).
• operation (e.g., mov).
• operands (e.g., ax,$1).

The easiest way to do it is to ensure that tokens are easily distinguishable.

That's why I made the rule that labels had to begin with : - it made the analysis of the line so much easier. The process for handling a line was:

• strip off comments (first ; outside a string to end of line).
• extract label if present.
• first word is then the operation.
• rest are the operands.

You can easily insist that different operands have special markers as well, to make your life easier. All this is assuming you have control over the input format. If you're required to use Intel or AT&T format, it's a little more difficult.

The way I approached it is that there was a simple per-operation function that got called (e.g., doJmp, doCall, doRet) and that function decided on what was allowed in the operands.

For example, doCall only allows a numeric or label, doRet allows nothing.

For example, here's a code segment from the encInstr function:

private static MultiRet encInstr(
    boolean ignoreVars,
    String opcode,
    String operands)
{
    if (opcode.length() == 0) return hlprNone(ignoreVars);
    if (opcode.equals("defb"))  return hlprByte(ignoreVars,operands);
    if (opcode.equals("defbr")) return hlprByteR(ignoreVars,operands);
    if (opcode.equals("defs"))  return hlprString(ignoreVars,operands);
    if (opcode.equals("defw"))  return hlprWord(ignoreVars,operands);
    if (opcode.equals("defwr")) return hlprWordR(ignoreVars,operands);
    if (opcode.equals("equ"))   return hlprNone(ignoreVars);
    if (opcode.equals("org"))   return hlprNone(ignoreVars);

    if (opcode.equals("adc"))   return hlprTwoReg(ignoreVars,0x0a,operands);
    if (opcode.equals("add"))   return hlprTwoReg(ignoreVars,0x09,operands);
    if (opcode.equals("and"))   return hlprTwoReg(ignoreVars,0x0d,operands);

The hlpr... functions simply took the operands and returned a byte array containing the instructions. They're useful when many operations have similar operand requirements, such as adc,addandand` all requiring two register operands in the above case (the second parameter controlled what opcode was returned for the instruction).

By making the types of operands easily distinguishable, you can check what operands are provided, whether they are legal and which byte sequences to generate. The separation of operations into their own functions provides for a nice logical structure.

In addition, most CPUs follow a reasonably logical translation from opcode to operation (to make the chip designers lives easier) so there will be very similar calculations on all opcodes that allow, for example, indexed addressing.

In order to properly create code in a CPU that allows variable-length instructions, you're best of doing it in two passes.

In the first pass, don't generate code, just generate the lengths of instructions. This allows you to assign values to all labels as you encounter them. The second pass will generate the code and can fill in references to those labels since their values are known. The ignoreVars in that code segment above was used for this purpose (byte sequences of code were returned so we could know the length but any references to symbols just used 0).

Answer 2

Not to discourage you, but there are already many assemblers with various bells and whistles. Please consider contributing to an existing open source project like elftoolchain.