Disassembling self-modifying code

Question

I've been just wondering - how does one disassemble self-modifying binary? I guess you can't use olly or IDA because they are static disassemblers, correct? What happens with disassembler if you jump into middle of an instruction? And how does one analyze metamorphic engines?

Answer 1

Both OllyDbg and IDA are not only static analyzers, they can both also run the code. IDA can also run your code remotely and as far as I know, it can even do embedded debugging. Of course if you "break" the program execution at some point, and take a look at the disassembly, it will (in both programs) reflect the current state of the program, including any modifications it has made on itself.

What happens with disassembler if you jump into middle of an instruction?

According to my experience, both the mentioned disassembler can handle this situation. For example, see here, how OllyDbg manages it, here is a screenshot when EIP is 00892C0E:

And when I make EIP = 00892C0F, which is in a middle of an instruction:

as you see, it simply re-disassembles the instruction, making it a different (but still valid) opcode.

And how does one analyze metamorphic engines?

Just as any other code. The tricks you are mentioning (jumping into a middle of a instruction, modifying itself) were mainly popular a while ago, when the disassemblers and debugger weren't as smart as they are now.

Of course static analysis can be very hard, but you can of course still analyze binaries completely offline and decipher the "morphing" (in your mind) to get the idea of what the code will do. But when you can use a debugger live, then you simply see what the code is up to.

Of course, all of this is an endless race between people writing the code and people analyzing it. Who wins depends on who gives up sooner.