What is to prevent a user process from using the trap instruction independently?

Question

The stub routine for a system call switches to kernel mode via a trap instruction. What is to prevent a user process from using the trap instruction independently to switch to kernel mode,and then executing malicious code after the switch?

Answer 1

The trap instruction doesn't simply switch the CPU mode from user to kernel; it actually does the following:

It switches the mode and JUMPS the CPU instruction pointer to a specified interrupt handler that is in the kernel, so technically once you call the trap instruction, your code won't be the one that's getting executed.

Let's take the x86 + FreeBSD as an example:
In all operating systems, there's something called interrupt vector which is basically a list of interrupts handlers, if an IO happens, interrupt handler 0xwhatever goes on, if you want to trap the CPU into kernel mode, interrupt handler 0x80 should be called; how does it get called?
int 0x80
Once this is done, CPU will go execute the handler #80(roughly speaking), which in turn will read some certain registers to determine what to do next(typically, you set the type of service you want the kernel to do for you and the parameters in registers).

So here, the only way you can switch to kernel mode is through the int instruction(there are instructions that switch too, in a slightly different manner but the concept holds), you have to specify a certain interrupt so the CPU will jump to the appropriate code and then that code, which is part of the kernel, will do the job it's supposed to do and return to your code AFTER returning mode to user-mode; you can just call int without any number but that won't just transfer the mode into kernel, it'll by default call interrupt handler 0, whatever that may be.

Other operating systems/CPUs would use different instructions but generally, the mechanism is the same.