How to pass parameters to Linux system call?

Question

I'm a college student studying OS.

I'm trying to add my own system call in Linux kernel, and something is going wrong.

My environment is stated below:

• Linux Kernel v.4.19.1
• 64-bit Ubuntu LTS 18.04.1 with Intel Core i5-4210M CPU on Oracle VirtualBox 5.2.18
• 64-bit Windows 10 Home 1803 as a host machine

Since I'm working on x86_64 machine, I started with arch/x86/entry/syscalls/syscall_64.tbl. In Linux kernel v.4.19.1, the last entry is

334     common     rseq             __x64_sys_rseq

so I added those three lines below it.

335     common     my_syscall_0     sys_my_syscall_0
336     common     my_syscall_1     sys_my_syscall_1
337     common     my_syscall_2     sys_my_syscall_2

Second, I added prototypes of functions in include/linux/syscalls.h.

asmlinkage int sys_my_syscall_0(void);
asmlinkage int sys_my_syscall_1(int);
asmlinkage int sys_my_syscall_2(int, int);

Third, I created a new file kernel/my_syscall.c and added implementations of functions.

asmlinkage int sys_my_syscall_0(void)
{
    printk("my_syscall_0\n");
    return 0;
}

asmlinkage int sys_my_syscall_1(int a)
{
    printk("my_syscall_1 : %d\n", a);
    return 0;
}

asmlinkage int sys_my_syscall_0(int a, int b)
{
    printk("my_syscall_2 : %d, %d\n", a, b);
    return b;
}

Then, I added my_syscall.o in kernel/Makefile to compile kernel/my_syscall.c as well.

obj-y = fork.o exec_domain.o panic.o \
        cpu.o exit.o softirq.o resource.o \
        sysctl.o sysctl_binary.o capability.o ptrace.o user.o \
        signal.o sys.o umh.o workqueue.o pid.o task_work.o \
        extable.o params.o \
        kthread.o sys_ni.o nsproxy.o \
        notifier.o ksysfs.o cred.o reboot.o \
        async.o range.o smpboot.o ucount.o \
        my_syscall.o

After compilation with make-kpkg and dpkg command, I made a test program.

#include <stdio.h>
#include <unistd.h>
#include <sys/syscall.h>
int main()
{
    printf("1 : %d\n", syscall(335));
    printf("2 : %d\n", syscall(336, 1));
    printf("3 : %d\n", syscall(337, 2, 3));
    return 0;
}

However, the result is weird. dmesg shows me some huge numbers that does not make sense at all.

# dmesg
...
my_syscall_0
my_syscall_1 : 1111490392
my_syscall_2 : 1111490392, 1111490392

It seems like it is changing every time I execute the program.

Obviously, there's some problem in passing parameters. However, strace command shows me that the values are passed well.

# strace ./syscall
...
syscall_0x14F(0x7ffd21866538, 0x7ffd21866548, ....) = 0
...
syscall_0x150(0x1, 0, 0, 0, 0x7f9e1562f3a0, ....) = 0
...
syscall_0x14F(0x2, 0x3, 0x7f9e1562f3a0, ....) = 0
....

Brief

1. I made simple system calls.

2. Passing parameter to them are not working as the way they should be.

3. Question : Is there any problem in steps above, in order to add new system call?

4. Question : Is there any issues that I should aware when passing parameters to system call?

Thank you in advance.

Answer 1

You need to tell the build system that your system call requires 2 arguments and that they are of type int. This is so that the scripts that are part of the build system will generate appropriate wrappers for casting the arguments into the type you require. Instead of defining the actual handler like you did, you should use -

SYSCALL_DEFINE2(my_syscall_2, int, a, int, b) // Yes, there is a comma between the types and the argument names
{
    printk("my_syscall_2 : %d, %d\n", a, b);
    return b;
}

SYSCALL_DEFINEx is defined in linux/include/linux/syscalls.h.

You can look at an example in linux/fs/read_write.c

Answer 2

I found the solution. As @Ajay Brahmakshatriya answered, I should use SYSCALL_DEFINEx macro. And also, I should modify arch/x86/entry/syscalls/syscall_64.tbl as well.

Here's final summary.

How to add new system calls

First, modify arch/x86/entry/syscalls/syscall_64.tbl : add those lines below.

335     common     my_syscall_0     __x64_sys_my_syscall_0
336     common     my_syscall_1     __x64_sys_my_syscall_1
337     common     my_syscall_2     __x64_sys_my_syscall_2

Second, modify include/linux/syscalls.h : add those lines below.

asmlinkage long sys_my_syscall_0(void);
asmlinkage long sys_my_syscall_1(int);
asmlinkage long sys_my_syscall_2(int, int);

Third, create a new file for implementation. For my case, kernel/my_syscall.c.

#include <linux/syscalls.h>
#include <linux/kernel.h>

SYSCALL_DEFINE0(my_syscall_0)
{
    printk("my_syscall_0\n");
    return 0;
}

SYSCALL_DEFINE1(my_syscall_1, int, a)
{
    printk("my_syscall_1 : %d\n", a);
    return 0;
}

SYSCALL_DEFINE2(my_syscall_2, int, a, int, b)
{
    printk("my_syscall_2 : %d, %d\n", a, b);
    return b;
}

Fourth, add created file to Makefile in its directory. For my case, kernel/Makefile.

...
obj-y = fork.o exec_domain.o panic.o \
        cpu.o exit.o softirq.o resource.o \
        sysctl.o sysctl_binary.o capability.o ptrace.o user.o \
        signal.o sys.o umh.o workqueue.o pid.o task_work.o \
        extable.o params.o \
        kthread.o sys_ni.o nsproxy.o \
        notifier.o ksysfs.o cred.o reboot.o \
        async.o range.o smpboot.o ucount.o \
        my_syscall.o
...

Finally, compile and install the kernel. Now, you would be able to see new system calls working well.

#include <stdio.h>
#include <unistd.h>
#include <sys/syscall.h>

int main()
{
    printf("1 : %d\n", syscall(335));
    printf("2 : %d\n", syscall(336, 1));
    printf("3 : %d\n", syscall(337, 2, 3));
    return 0;
}

dmesg command shows me that system calls are working well.

# dmesg
my_syscall_0
my_syscall_1 : 1
my_syscall_2 : 2, 3