Are assembly language programs restricted by kernel modes as well?

Question

If I were to write an assembly language program, am I allowed to break the security mechanisms of an OS - accessing other processes' address space, accessing kernel address space, etc?

Edit :

If I write an assembly language program which is loaded in memory by an OS supplied loader, how would the OS intercept my program while it tries to access protected memory address spaces? My assumption is this protection is built into the system calls and libraries supplied by an OS, and if you don't use those system calls and libraries, you're allowed to break OS policies.

Answer 1

For most architectures (including x86), protection is built-in into the hardware.

For x86, there are two techniques for protecting the address space of the kernel and the other programs: segmentation and paging. Most modern operating systems (including Windows, OS X, Linux, BSD, etc) use paging as the main technique, so I'll talk about it.

Paging involves having page directories and page tables, which describe the properties of each virtual page: physical address, permissions, etc. Page directories and page tables are mapped in kernel space and therefore can't be edited by user processes.

When the kernel switches to a user process, it restores the page structures of the user process so it only has permission to access its own mapped pages. When it tries to access some unmapped address, a page fault will be raised. If memory contents at that address had been swapped away, they will be unswapped and the process will continue without noticing anything (apart from a small delay maybe). If that memory address hadn't been allocated and therefore not swapped away, the program will simply terminate.

Note: It's also not true that only assembly language can cause unsafe behaviour. Try to write to a random address in C and you will see that it will most likely crash immediately. Why? Because the OS will detect the address space violation through the page fault. In case it doesn't crash, it will mean you will have written to some memory belonging to its own address space.

Answer 2

Not if your program is run as an OS-controlled process, on a machine that provides adequate process isolation (typically address space management protection) in hardware. (Well, you can write instructions that try to break security, but under a well-designed OS with such hardware all they will do is trap when executed).

If the hardware you run on does not have such protection capabilities, then you are running in a space shared with the OS and other applications, by convention in a cooperative way. That means you could touch and damage the OS and other applications, but you should not. Whether you choose to violate that commitment is up to you; normally, it doesn't make you friends with community of users of that type of hardware.

Answer 3

The protection is built into the hardware. Every single load/store (and code fetch) is checked against the page tables (using a TLB to cache page table entries).

Also, privileged instructions like x86's LIDT fault if run by a user-space process. (The exceptions section of that manual entry says it raises #GP(0) "If the current privilege level is not 0").

If all it took to get root on a machine was a little bit of hand-written asm, secure multi-user OSes would be impossible to implement. Since they exist and are actually quite common, we know your premise must be wrong. (Linux, Unix, OS X, Windows are all designed to be secure in this way. They've all had implementation bugs that allow attackers to run arbitrary kernel code, but that's not the point.)