Why does this code enable me to detect a debugger?

Question

Why the following assembly code is an anti-debugging tool?

l1:
call l3
l2:
;some code
l3:
mov al, 0c3h
mov edi, offset l3
or ecx, -1
rep stosb

I know that C3h is RETN and I know that stobs writes the value in al as opcode according to the offset in edi and it is done for ecx times because of rep.

I am also aware the fact that stobs and stosw will run if they were pre-fetched on intel architecture as their original format.

If we run the program in debugged mode the pre-fetch is irrelevant and the l2 label will run (because it is single-step) otherwise if there is no debugger it will be ping-pong between l1 and l3 am I right?

Answer 1

When program is debugged (i.e. single step) prefetch queue is flushed at each step (when interrupt occurs). However, when executed normally that will not happen to rep stosb. Older processors didn't flushed it even when there was memory write to the cached area, in order to support self-modifying code that was changed except rep movs and rep stosb. (IIRC it was eventually fixed in i7 processors.)

That's why if there is a debugger (single step) code will execute correctly and when rep stosb is replaced by ret l2 will be executed. When there is no debugger rep stosb will continue, since ecx is the biggest possible it will eventually write somewhere it is not supposed to write and an exception will occur.

This anti-debugging technique is described in this paper.