What are "non-virtualizable" instructions in x86 architecture?

Question

Before the advent of hardware assisted virtualization there were instructions that could not be virtualized due to various reasons. Can somebody please explain what those instructions are and why they cannot be virtualized?

Answer 1

To virtualize an ISA, certain requirements must be met. Popek and Goldberg used something like the following:

A machine has at least two modes (a) user mode and (b) system mode. Typically, applications run in user mode and the operating system runs in system mode. In system mode, the code/program can see and manipulate the machine without restrictions. In user mode, the code/program has some limitations in what it can do, e.g. it can't access all of the machine's memory without acquiring permission first.

Instructions are either (a) privileged or (b) not privileged. Privileged instructions trap when executed in user mode. Trapping means that the machine is forced into system mode whereby it executes some code of the operating system to deal with the situation. In a sense, they alert the operating system when executed.

Instructions can also be either (a) sensitive or (b) not sensitive. Sensitive instructions modify part of the machine's resources, or, exhibit different behaviour depending if they are executed in user mode or system mode.

When virtualizing an ISA, it is important that the virtual machine monitor (VMM) can detect, and handle smoothly, any attempt of the program or guest operating system to modify the machine's resources. It must be able to see when sensitive instructions are being executed. To do this, all of the sensitive instructions need to be privileged and thus trap when executed. When trapped, we can enter system mode and call code from the VMM to handle the resource modification.

The problem is that not all of X86's sensitive instructions are privileged instructions. This means that resource modification can occur without the VMM seeing and handling it which can be dangerous. Alternatively, it could mean executing an instruction within the guest operating system in user mode and seeing a different effect than having executed it in system mode. According to this paper there are seventeen instructions in x86 that are sensitive but are not privileged. One example is POPF when has different semantics depending on the machine's mode.