What is CPU kernel/privileged mode, and how is it guarded by the OS?

Question

In Kernel mode, the executing code has complete and unrestricted access to the underlying hardware. It can execute any CPU instruction and reference any memory address

Any example? Like what instructions, and what memory address?(x86 preferably). As for memory, does it mean like the high 1GB address(32 bit) reserved for kernel? Besides, who decides this? Is it the CPU, or OS? If it's the OS, how does the CPU know what part of memory is considered reserved(maybe some bit in the page table)? And how is memory accessed in kernel mode, still by virtual address?

As for entering kernel mode, it is done by certain instructions like int right? I understand the part the all the system calls are heavily guarded, but what if I write a program in assembly code, which includes the int instructions? How is that guarded? I mean when that program is compiled, and eventually I got an executable file, the int instruction has to be in the '.text' section, coz the compiler won't do anything about it right? So what does OS do about it when it runs? Or what's in the '.text' section indeed? Is it THE instructions that are recognised by CPU, or maybe it's still some intermediate format, that would be 'translated' by the OS, so the OS would have a chance to 'reject' certain instructions like 'int', or whatever other 'restricted' instructions before it runs by CPU?

Update: For anyone who's interested, here's a great explanation(at least for me) from quora: https://www.quora.com/What-is-CPU-kernel-mode-and-how-is-it-guarded-by-the-OS/answer/Mostafa-Abd-El-Aziz?snids=937993065&nsrc=1&filter=all

Answer 1

The reason we have kernel mode is protect processes from each other and themselves. (At the risk of oversimplification) each process sees its address space divided between a user and kernel region. The user region belongs solely (with some exceptions) to that process. All processes share the same system region.

Restricting access to the system region to kernel mode protects it from rogue processes running in user mode.

All processes need to communicate with the kernel. The process for doing that is either an interrupt or an exception. Both are handled the same way, the difference is how they are triggered.

The kernel defines paths into itself through a dispatch table (various names are used). Exceptions and interrupts have numbers that are indices into the table contain addresses of functions to handle a particular event or service.

Example:

If you want to do I/O, your fprintf statement eventually translates into a user mode wrapper function (system service) that sets up the stack and registers in a defined manner. That system service triggers an exception. The dispatch table sends it to the correct kernel mode handler for handling writes to disk (usually writes in general).

The kernel mode handler needs to carefully validate all the parameters sent to it. Are the register values correct? Are all references to memory valid and in the user region? Any validation holes here can call the system to be corrupted.

The the kernel mode function causes the data you want to write to be sent to the disk at the correct stop.

To your questions:

As for entering kernel mode, it is done by certain instructions like int right?

Yes. Also exceptions, such as divide by zero, page faults. IMHO, the Intel INT instruction is an exception generator, not a software interrupt as it is usually described. The three things that get you into kernel mode are:

1. Interrupts (assynchonous events)
2. Traps (INT causes a trap)
3. Faults (Exceptions that can be corrected and allow reelection of the instruction that caused them)

I understand the part the all the system calls are heavily guarded, but what if I write a program in assembly code, which includes the int instructions?

Your application gets routed through the dispatch table. The function that gets called there should catch an invalid call.

How is that guarded? I mean when that program is compiled, and eventually I got an executable file, the int instruction has to be in the '.text' section, coz the compiler won't do anything about it right? So what does OS do about it when it runs? Or what's in the '.text' section indeed? Is it THE instructions that are recognised by CPU, or maybe it's still some intermediate format, that would be 'translated' by the OS, so the OS would have a chance to 'reject' certain instructions like 'int', or whatever other 'restricted' instructions before it runs by CPU?

It is guarded by 1) forcing you to go to the handler in the dispatch table and 2) the handler doing proper validation.

Answer 2

its all in the manual.

Say for example the video memory/registers. or say for example some other program that is also running on the computer that has some memory for instructions and data. your program should not be allowed to touch any of that directly the kernel has permission and can, good, bad or otherwise, touch any of it, but your application can only touch your memory.

How does this work? depending on the processor family and other things but generally there is an mmu which allows your program to use a virtual address, that virtual address is fed through the mmu which is logic that looks up things in tables based on the address you are using, be it the address to the next instruction, the address for a variable in memory or whatever. the mmu along with translation from virtual addresses to physical addresses also the tables contain permissions and cached or not cached information. So if you access an address that the mmu table shows is not yours then a fault occurs and depending on the hardware (processor) and the operating system something happens, it may just give you garbage or 0xFFs and let you keep going off into the weeds or it may kill your application all together or somewhere in between. on an x86 system there is something similar for I/O space as well, giving you permission or not to I/o addresses.

Now the faults are desireable, for example you have heard of virtual memory perhaps. swap or swap space or whatever. say all of the applications running are requiring more ram than the system has, well the latest application gets its turn it needs some ram. some other applications ram is chosen, that ram is written to a swap file or hard disk or some other media basically, and then the mmu table for the old application is marked such that it will fault for you it is marked so you get a virtual address to that physical ram. when the other application comes along and wants to access that ram that was once his, he gets a fault the operating system gets called, and remembers that it had swapped out that applications memory to disk, so it now finds some ram from someone else, takes it, saves that to disk, and from disk pulls the current applications memory patches its mmu table AND LETS IT KEEP RUNNING, this kind of thing is happening constantly. they also do this to virtualize peripherals or allow multiple applications to think they are using a peripheral but are all really sharing and nobody is really touching the peripheral just touching virtual ones that each access causes a fault which causes the os to simulate the peripheral. there are then endless things you can do.

the mmu also allows for programs to all think they are in the same address space, so say all programs are compiled to have an entry point of 0x8000 and ram is linear from 0x00000 to how much you need or can address. how would that be possible for every program to have the same address space? the reality is that none of them have that address space, what they think is a linear address space from address zero to N is really chopped up by the mmu into various sized chunks each chunk can be in a completely different place in ram. maybe 0x8000 to 0x10000 is at physical address 0xD0008000 right now and 0x10000 is at physical address 0x22000000 right now, that is perfectly fine and not uncommon. As each program is launched the operating system knows what physical address spaces are available as it is managing it, the program says it wants N bytes so the operating system pieces together that ram, unlike the old days pre-mmu you dont necessarily have to move other stuff up or down to make a hole N bytes big, you just need to find some number of sub N byte chunks that fit into the mmu table such that you end up with N total bytes, you then fill that space with the program and let the application start. So the .text for each of your programs built by the same compiler for the same operating system will put .text in the same place and .data ideally, and so on, or at least it has to conform to that operating systems rules. it is the execution mode or the virtual id of the running application or some processor specific mechanism that determines whether you are a protected or unprotected program. on some I would assume most systems even the kernel code has to go through the mmu but the kernels id or mode or whatever is marked as having permission to everything. and not required but it may be that for peripheral access the virtual address equals the physical address, but it might not you might still have to do some sort of kernel api call to get a virtual address/pointer to some physical address (similar to mmap for applications).

Who decides. The operating system decides and implements whatever is required based on the rules for that cpu and other hardware. it is usually a combination of cpu and operating system specific how you basically switch from application to kernel or basically make "system calls" from your application. when you read the manual it will spell this out in some form. ARM for example is much easier for example to learn this on first then go for something more convoluted...The modes are spelled out for you in the programmers model chapter and the svc instruction (has an alias name that I cant think of right now) tells you that it switches the processor to that mode which hits a handler in that mode which is basically the kernel/operating system which implements the call and returns switching mode back (assuming it is a call that returns and not the application exiting or being killed due to an error). Likewise a data abort or prefetch abort is clearly documented on what mode it enters and so on. user mode restrictions are defined and you start to see that the user mode code is really trapped in a safe sandbox that it cant get out of.

so it is not a case of this huge range above this physical address is protected and below is fair game, ALL of your APPLICATION address space is managed for you by the operating system and enforced by the cpu and associated hardware, all of it is virtual addresses which are probably rarely mapped to physical addresses. the kernel may or may not play in physical addresses but its execution mode or id has more permissions than yours. Your visitors badge may limit you to a waiting room or cafeteria perhaps but other peoples badges will grant them access deeper into the bowels of the corporation/building, all managed by someone with god like powers that program the system as to which doors your badge or his badge or her badge will open depending on their requirements and trust.