Implementing a syscall on real-time Debian Wheezy

Question

For educational purposes, I want to implement a system call in Debian Wheezy. I wish to implement it on the kernel that comes in the linux-image-3.2.0--rt-amd64 package. Here is an overview of what I have tried:

To get the kernel source:

apt-get source linux-image-3.2.0-4-rt-amd64

From that, I get the following files/directories the directory I executed in:

linux_3.2.41.orig.tar.xz
linux_3.2.41-2+deb7u2.dsc
linux_3.2.41-2+deb7u2.debian.tar.xz

as well as:

linux_3.2.41

which contains the source code for the kernel.

Then, to make the necessary changes in order to add the system call, I basically followed this page: How to write system calls on debian/ubuntu

The following is a condensed version of the instructions given there modified to reflect the changes I made.

+File 1: linux-x.x.x/vpart_syscalls/vpart_syscalls.c

#include <linux/linkage.h>
#include <linux/kernel.h>

asmlinkage long insert_partition(char*dest, const char* src)
{
    printk("<--- the syscall has been called!");
    return 0;
}

• File 2: linux-x.x.x/vpart_syscalls/Makefile. Create a Makefile within the same test directory you created above and put this line in it:

  obj-y := vpart_syscalls.o

• File 3: linux-x.x.x/arch/x86/kernel/syscall_table_32.S. Now, you have to add your system call to the system call table. Append to the file the following line:

  .long insert_partition

• File 4: linux-x.x.x/arch/x86/include/asm/unistd_32.h

In this file, the names of all the system calls will be associated with a unique number. After the last system call-number pair, add a line

#define __NR_insert_partition 349

Then replace NR_syscalls value, stating total number of system calls with (the existing number incremented by 1) i.e. in this case the NR_syscalls should've been 338 and the new value is 339.

#define NR_syscalls 350

• File 5: linux-x.x.x/include/linux/syscalls.h

Append to the file the prototype of our function.

asmlinkage long insert_partition(int lenTicks, int vpid);

just before the #endif line in the file.

• File 6: Makefile at the root of source directory.

Open Makefile and find the line where core-y is defined and add the directory test to the end of that line.

core-y += kernel/ mm/ fs/ test/ vpart_syscalls/

I then proceeded to build the kernel in a different fashion than is described there:

make localmodconfig
make menuconfig (making no changes)


make-kpkg clean
fakeroot make-kpkg --initrd --append-to-version=+tm kernel_image kernel_headers
cd ..
dpkg -i linux-image-3.8.*
dpkg -i linux-headers-3.8.*

The kernel that is installed boots fine. I made the following c program to test the syscall:

#include <stdio.h>
#include <linux/unistd.h>
#include <sys/syscall.h>

int main(){
    printk("Calling the new syscall!\n");
    int ret = 100;
    ret = syscall(349, 1, 2);
    printf("call return value: %i\n", ret);
    return 0;

}

When I compile and run this program, I get a return value of -1. I check the messages using dmesg and there is no evidence of my printk being called..

If anyone knows where my problem is I would be really really happy! I should say I am not too experienced at changing and building the kernel, but I have learned a lot about it. I read Robert Loves book - linux kernel development and several guides on the webs.

Answer 1

I think, the steps 3 and 4 may be incorrect for 64-bit kernels:

File 3: linux-x.x.x/arch/x86/kernel/syscall_table_32.S. 
File 4: linux-x.x.x/arch/x86/include/asm/unistd_32.h

There are two files here: http://lxr.linux.no/linux+v3.2.41/arch/x86/kernel/

syscall_64.c    668 2008-12-24 14:26:58 -0800   
syscall_table_32.S  8659    2012-01-04 14:55:50 -0800

First one defines syscall table contents for 64-bit mode using C file and macro-cheating with unistd_64.h

#define __SYSCALL(nr, sym) [nr] = sym,

const sys_call_ptr_t sys_call_table[__NR_syscall_max+1] = {
....
#include <asm/unistd_64.h>
};

Where asm/unistd_64.h is

 #define __NR_read                               0
 __SYSCALL(__NR_read, sys_read)

and so on.

And second one, which you changed - is for 32-bit mode and written using asm file and labels (.long sys_call_name).

So, you defined syscall for 32-bit mode and you are using linux-image-3.2.0-4-rt-amd64 which is basically for " 64-bit PCs".

I think you compiled your test program as gcc test.c, which defaults to 64-bit mode. You can try -m32 option of gcc: gcc -m32 test.c to get 32-bit application (this will only work if you have correct cross environment for 32-bit builds) or compile this test on some 32-bit linux.

Or the other choice is to make step "4a": edit arch/x86/include/asm/unistd_64.h to add two lines:

 #define __NR_insert_partition                               YOUR_NUMBER
 __SYSCALL(__NR_insert_partition, insert_partition)

I'm not sure where and how NR_syscalls for 64bit is defined. It may be generated during build.