syscall error handling inside the linux kernel

Question

I'm writing a new syscall inside the linux kernel. I want to implement an error handling system to return to the user who uses the sycall a code which describes the type of the error. I'd like to know how this is done inside the kernel, since I want to follow the standard. I want to let the user read the errno variable, or something equivalent, to understand the error that happened. Thank you.

P.S. I'm using the latest versione of linux available.

Answer 1

Much of your task is taken care of automatically by libc and the low-level kernel syscall handler (the part written in assembly). The kernel convention for handling error codes is to return a negative error constant, something like -ENOMEM. A zero or a positive number indicates success. This is used throughout the kernel.

If you've defined the new sycall entry point static asmlinkage long my_new_syscall (int param) {...}, it just needs to return -ENOMEM (or the like). If you invoked the system call by using libc's syscall(nr, param), then on an error it will return -1 and ENOMEM (in the positive) will be in errno.

There are many error codes defined in include/asm-generic/{errno,errno-base}.h that you can use (like ENOMEM). If none of those fit your purpose, you can add your own error number to those files, but be aware that you'll also need to modify the user-space-visible kernel headers to show the same number, so it will be more difficult to set up a system to use your modifications. Don't do this if you don't have to.