Adding a new system call to Linux that requires root privileges

Question

I'm trying to add a system call to the linux kernel(version:3.10.91) that requires root privileges to run.

You can see my attempt below:

#include <linux/kernel.h>
#include <linux/linkage.h>
#include <linux/sched.h>
#include <asm/current.h>
#include <asm/errno.h>

asmlinkage long sys_set_casper(pid_t pid, int value)
{
    struct task_struct *process;
    
    if (!capable(CAP_SYS_ADMIN))
        return -EPERM;
    
    //valid values are 0,1,2,3
    if (value != 0 && value != 1 && value != 2 && value != 3 )
        return -EINVAL;
    
    process = find_task_by_vpid(pid);
    if (process == NULL)
        return -ESRCH;

    //modify the casper field accordingly   
    process->casper = value;
    return 0;   
}

Casper is just a task descriptor that I added. Basically, I want the process to be hidden(not visible to ps,pstree,top etc.) when the casper value is 1. Kernel recompiles jut fine(and I also did the necessary changes in base.c etc.).

I tried testing my system call with the following code, test.c:

#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#define NR_set_casper 351

int main(int argc, char **argv)
{
    long y;
    printf("PID of current process: %d\n\n", getpid());
    
    printf("Call set_casper system call to set flag 1\n");
    y = syscall(NR_set_casper, getpid(), 0);
    printf("Return value of set_casper system call: %ld\n", y);
    if (y < 0)
    {
        printf("set_casper system call failed. Run with sudo\n");
        return EXIT_FAILURE;
    }

    return 0;
}

I compile and run as follows:

gcc test.c

sudo ./a.out

The output is:

PID of current process: 3852

Call set_casper system call to set flag 1
Return value of set_casper system call: -1
set_casper system call failed. Run with sudo

The weird thing is, even after removing the sudo control lines:

 if (!capable(CAP_SYS_ADMIN))
        return -EPERM;

and recompiling the kernel, I still got the same error.

Basically, why my sys_set_casper function returns -1 ?

EDIT

I've added this: 351 i386 set_casper sys_set_casper to arch/x86/syscalls$ gedit syscall_32.tbl

However, my system is 64 bit. Could this be the problem?

Answer 1

The problem was, as others stated in comments, that I wasn't calling the system call at all. I simply added my call to the 64 bit system call table and tried everything again and it worked.