How to create an x86 binary with 'fake' instructions in it

Question

I'm working on a project where I'd like to add some custom op codes to x86 and run them in QEMU. I've sort of figured out how to modify the code generation in QEMU to take an assembly instruction with a 'fake' opcode and to do something with it in QEMU.

However, the part I'm having trouble with is how I would actually go about creating a binary with a fake instruction in it. The only way I've thought about is adding in some db statements and then just writing the instruction manually. For example:

xor EAX, EBX
db 0xf1,0x32,0x55,0x00
mov EBX, EAX

(assume the db has enough bytes to be an actual instruction). Would this actually compile a binary where the bytes I defined in the second line are treated as an instruction?

Is there some more elegant approach to this? Since I'll be modifying QEMU to support these changes, I'm not really bound by the instruction format -- I just need to get the OP code to be recognized by the QEMU's code generator and the rest I can formulate however I want.

Answer 1

The approach of using db is standard procedure when a new model of CPU comes out for which there are no immediate updates for current tools to support additional addressing modes and/or instructions.

Note that it is now rather challenging to find holes in the x86 instruction set for defining new instructions. Check Intel's latest CPU architecture documents: there is always a set of tables in an appendix showing opcode mapping. That would be the easiest way to find an unused instruction opcode sequence.

Answer 2

The db is fine and will emit the given sequence of bytes immediately after the first instruction and before the last one.

You might consider creating a bunch of assembler macros for your new instructions, that'll take the relevant arguments and emit your magic bytecodes.