Changing linux kernel system call number

Question

I wanted to build my own custom kernel with a different syscall table. (same syscalls but in different position/numbers)

I was working on kernel 3.2.29.

Changing the kernel was quite easy:

1) changing the syscall position in ‫‪arch/x86/kernel/syscall_table_32.S‬‬

2) changing the syscall macro number in arch/x86/include/asm/unistd_32.h

3) compiling and installing the new kernel

I switched the syscalls around: sys_open took the place and number of sys_read, and vice versa.

I figured that if I compile glibc with the modified kernel headers, I could have a running system, but unfortunately, it wasn't enough and my system won't boot.

Am I missing something? What else do I need to do in order to have a running system?

---

The steps I have taken are:

1) building and installing the kernel as described in my question

2) extracting the new kernel headers using make headers_install INSTALL_HDR_PATH=[path]

3) building glibc with the parameter --with-headers=[path/include]

4) I used a live cd to access the file system externally in order to install the new glibc, using the make install install_root=[the original file system] (so the system won't break during the install)

I hope that the new glibc was built properly, but I am not sure.

After that, when booting the system, the boot stops in the (initrafms) shell screen: I guess I need to rebuild the initrd, but how do I compile it according to the new syscall table?

Answer 1

You will have to rebuild everything. Even if all your binaries are dynamically linked, it is possible that the old syscalls were inlined into the binary because many of the C functions are just return syscall(__NR_somecall,...).

You could do this manually, but it could be difficult to keep the toolchains straight unless you use a cross compilable toolchain like buildroot, aboriginal or similar. Pick whichever best suits you (I prefer Rob Landley's aboriginal - http://landley.net/aboriginal/ )

Then to make your initrd just expand the old one using {z,bz,xz}cat oldinit.rd |cpio -id; rm oldinit.rd. Replace the old kernel modules, libs and binaries with the new and cpio and compress it back (cpio needs the -H newc option) ... or now you can rebuild your kernel and point the initramfs to that directory, but wouldn't recommend that if your initrd may need changed frequently such as if for instance you were testing out a whole new syscall structure and having to debug a lot.