What enforces memory protection in an OS?

Question

In general, I know that a process can't write to a memory (in its addresses space) that has a protection that doesn't allow writing. But what checks whether the process can do this? Does any assembly instruction goes through the operating system? how does it work?

Answer 1

In most modern CPUs (Intel x86, most ARM flavors) it's the CPU itself that does the checking. The CPU stores, in one of the registers, an address of a data structure that specifies the layout of the memory ("page table") - specifically, which addresses are readable, which are writable, which are executable. Every memory accessing operation in the CPU is checked against the page table.

When a program tries to do something to a memory location that the respective page table entry does not allow, the CPU generates an exception (interrupt), and the OS gets control. Further actions depend on the OS. One common scenario involves the OS displaying an error message and terminating the faulty program. Not necessarily, though. For example, page swapping (writing memory out to a page file on a disk and reading back when needed) is implemented via the same mechanism.

The page table is maintained by the OS and is not (typically) visible to userland code. The relevant portions in the OS are hardware dependent.