order for encoding x86 instruction prefix bytes

Question

I know that x86 instructions can have a maximum of 4 bytes of prefixes, e.g Lock, rep, segment overrides etc.

Is there any particular order in which they should appear, in case multiple prefixes are used?

Answer 1

the order can be found in volume 2A of the Intel Software Developer's Manual.

In a nutshell:

• the F2 and F3 prefixes cancel each other out. The one that comes later has precedence.
• the 66 prefix is ignored if either F2 or F3 are used (as mandatory prefixes in a long instruction). This of course doesn't apply to rep movsw where both those prefixes are simply prefixes, not part of the opcode.
• the REX escape may not be followed by any other prefixes.
• the VEX escape may not be preceded by REX, 66, F2 or F3

for the rest, the order shouldn't matter.

Answer 2

Quote from Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 2A: Instruction Set Reference, A-M

Instruction prefixes are divided into four groups, each with a set of allowable prefix codes. For each instruction, it is only useful to include up to one prefix code from each of the four groups (Groups 1, 2, 3, 4). Groups 1 through 4 may be placed in any order relative to each other.