x86 TEST instruction not working?

Question

I've been banging my head against the wall figuring this out, and this is making no sense to me...

Why does this program enter an infinite loop?!

I thought you could use test to compare two values for equality, as shown here... why doesn't it work?

int main()
{
    __asm
    {
        mov EAX, 1;
        mov EDX, EAX;
        test EAX, EDX;
L:      jne L;
    }
}

Answer 1

Your expectation of what the TEST instruction does is incorrect.

The instruction is used to perform bit tests. You would typically use it to "test" if certain bits are set given a mask. It would be used in conjunction with the JZ (jump if zero) or JNZ (jump if not zero) instructions.

The test involves performing a bitwise-AND on the two operands and sets the appropriate flags (discarding the result). If none of the corresponding bits in the mask are set, then the ZF (zero flag) will be 1 (all bits are zero). If you wanted to test if any were set, you'd use the JNZ instruction. If you wanted to test if none were set, you'd use the JZ instruction.

The JE and JNE are not appropriate for this instruction because they interpret the flags differently.

---

You are trying to perform an equality check on some variables. You should be using the CMP instruction. You would typically use it to compare values with each other.

The comparison effectively subtracts the operands and only sets the flags (discarding the result). When equal, the difference of the two values is 0 (ZF = 1). When not equal, the difference of the two values is non-zero (ZF = 0). If you wanted to test if they were equal, you'd use the JE (jump if equal) instruction. If you wanted to test if they were not equal, you'd use the JNE (jump if not equal) instruction.

---

In this case, since you used TEST, the resulting flags would yield ZF = 0 (0x1 & 0x1 = 0x1, non-zero). Since ZF = 0, the JNE instruction would take the branch as you are seeing here.

tl;dr

You need to compare the values using the CMP instruction if you are checking for equality, not TEST them.

int main()
{
    __asm
    {
        mov EAX, 1
        mov EDX, EAX
        cmp EAX, EDX
L:      jne L          ; no more infinite loop
    }
}

Answer 2

Just reading this (my asm is very rusty) and this

JNE jumps on ZF (Zero Flag) = 0

TEST sets ZF = 0 If bitwise EAX AND EDX results in 1 and 1 if bitwise AND results in 0

If the result of the AND is 0, the ZF is set to 1, otherwise set to 0.

Therefore it jumps as 1 AND 1 results in 0 in ZF.

Seems logical yet counter intuative.

I think @A.Webb is right - it should probably be JNZ if you're using the TEST instruction as you are relying on the behavior of a bitwuse operation to set the zero flag whereas the SUB instruction would set the Zero flag as you need.

Answer 3

This is pretty simple. You obviously need to know what the instructions do, what processor state they read, and write. When in doubt, get a reference manual. The Intel x86 manuals are easy to find online.

Your specific program:

    mov EAX, 1;

moves the constant 1 to EAX. No other state changes occur.

    mov EDX, EAX;

copies the contexts of EAX into EDX, so it too contains the value 1.

    test EAX, EDX;

test computes the bitwise AND of two registers (did you check the reference manuals?), throws the answer away, and sets condition code bits based on the answer. In your case, the upper 31 bits of each register is zero, and'd produces zeros. The least significant bit is one in both registers; and'd produces a 1. The net effect is that the 32 binary value "one" is generated, and thrown away after the condition code bits are set. There is one condition code bit we care about for this program, and that's the "Z"(ero) bit, which is set if the last condition-code setting operation produced a full zero value. This test produced "one", so the Z bit is reset. I'll let you look up the other condition code bits.

 L:      jne L;

This is a "Jmp on Not Equal", e.g, it jmps if the Z bit is reset. For your program, Z is reset, the jmp occurs. After execution, the processor is at the same insruction, and sees another (the same jmp). The condition code bits aren't changed by a jmp instruction.

So... it goes into an infinite loop.

There are lots of synonyms for various opcodes supported by assemblers. For instance, "JZ" and "JE" are synonyms for the same instruction. Don't let the synonyms confuse.