Override libc functions called from another libc function with LD_PRELOAD

Question

I've a project aiming to run php-cgi chrooted for mass virtual hosting (more than 10k virtual host), with each virtual host having their own chroot, under Ubuntu Lucid x86_64.

I would like to avoid creating the necessary environment inside each chroot for things like /dev/null, /dev/zero, locales, icons... and whatever which could be needed by php modules thinking that they run outside chroot.

The goal is to make php-cgi run inside a chroot, but allowing him access to files outside the chroot as long as those files are (for most of them) opened in read-only mode, and on an allowed list (/dev/log, /dev/zero, /dev/null, path to the locales...)

The obvious way seems to create (or use if it exists) a kernel module, which could hook and redirect trusted open() paths, outside of the chroot. But I don't think it's the easiest way:

• I've never done a kernel module, so I do not correctly estimate the difficulty.
• There seems to be multiple syscall to hook file "open" (open, connect, mmap...), but I guess there is a common kernel function for everything related to file opening.

I do want to minimize the number of patchs to php or it's module, to minimize the amount of work needed each time I will update our platform to the latest stable PHP release (and so update from upstream PHP releases more often and quickly), so I find better to patch the behavior of PHP from the outside (because we have a particular setup, so patching PHP and propose patch to upstream is not relevant).

Instead, I'm currently trying an userland solution : hook libc functions with LD_PRELOAD, which works well in most cases and is really quick to implement, but I've encountered a problem which I'm unable to resolve alone. (The idea is to talk to a daemon running outside the chroot, and get file descriptor from it using ioctl SENDFD and RECVFD).

When I call syslog() (without openlog() first), syslog() calls connect() to open a file.

Example:

folays@phenix:~/ldpreload$ strace logger test 2>&1 | grep connect
connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
connect(1, {sa_family=AF_FILE, path="/dev/log"}, 110) = 0

So far so good, I've tried to hook the connect() function of libc, without success. I've also tried to put some flags to dlopen() inside the _init() function of my preload library to test if some of them could make this work, without success

Here is the relevant code of my preload library:

void __attribute__((constructor)) my_init(void)
{
  printf("INIT preloadz %s\n", __progname);
  dlopen(getenv("LD_PRELOAD"), RTLD_NOLOAD | RTLD_DEEPBIND | RTLD_GLOBAL |
                               RTLD_NOW);
}

int connect(int sockfd, const struct sockaddr *addr, socklen_t addrlen)
{
  printf("HOOKED connect\n");
  int (*f)() = dlsym(RTLD_NEXT, "connect");
  int ret = f(sockfd, addr, addrlen);
  return ret;
}

int __connect(int sockfd, const struct sockaddr *addr, socklen_t addrlen)
{
  printf("HOOKED __connect\n");
  int (*f)() = dlsym(RTLD_NEXT, "connect");
  int ret = f(sockfd, addr, addrlen);
  return ret;
}

But the connect() function of the libc still takes precedence over mine:

folays@phenix:~/ldpreload$ LD_PRELOAD=./lib-preload.so logger test
INIT preloadz logger
[...] no lines with "HOOKED connect..." [...]
folays@phenix:~/ldpreload$

Looking at the code of syslog() (apt-get source libc6 , glibc-2.13/misc/syslog.c), it seems to call openlog_internal, which in turn call __connect(), at misc/syslog.c line 386:

            if (LogFile != -1 && !connected)
            {
                    int old_errno = errno;
                    if (__connect(LogFile, &SyslogAddr, sizeof(SyslogAddr))
                        == -1)
                    {

Well, objdump shows me connect and __connect in the dynamic symbol table of libc:

folays@phenix:~/ldpreload$ objdump -T /lib/x86_64-linux-gnu/libc.so.6 |grep -i connec
00000000000e6d00  w   DF .text  000000000000005e  GLIBC_2.2.5 connect
00000000000e6d00  w   DF .text  000000000000005e  GLIBC_2.2.5 __connect

But no connect symbol in the dynamic relocation entries, so I guess that it explains why I cannot successfully override the connect() used by openlog_internal(), it probably does not use dynamic symbol relocation, and probably has the address of the __connect() function in hard (a relative -fPIC offset?).

folays@phenix:~/ldpreload$ objdump -R /lib/x86_64-linux-gnu/libc.so.6 |grep -i connec
folays@phenix:~/ldpreload$ 

connect is a weak alias to __connect:

 eglibc-2.13/socket/connect.c:weak_alias (__connect, connect)

gdb is still able to breakpoint on the libc connect symbol of the libc:

folays@phenix:~/ldpreload$ gdb logger
(gdb) b connect
Breakpoint 1 at 0x400dc8
(gdb) r test
Starting program: /usr/bin/logger 

Breakpoint 1, connect () at ../sysdeps/unix/syscall-template.S:82
82      ../sysdeps/unix/syscall-template.S: No such file or directory.
        in ../sysdeps/unix/syscall-template.S
(gdb) c 2
Will ignore next crossing of breakpoint 1.  Continuing.

Breakpoint 1, connect () at ../sysdeps/unix/syscall-template.S:82
82      in ../sysdeps/unix/syscall-template.S
(gdb) bt
#0  connect () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007ffff7b28974 in openlog_internal (ident=<value optimized out>, logstat=<value optimized out>, logfac=<value optimized out>) at ../misc/syslog.c:386
#2  0x00007ffff7b29187 in __vsyslog_chk (pri=<value optimized out>, flag=1, fmt=0x40198e "%s", ap=0x7fffffffdd40) at ../misc/syslog.c:274
#3  0x00007ffff7b293af in __syslog_chk (pri=<value optimized out>, flag=<value optimized out>, fmt=<value optimized out>) at ../misc/syslog.c:131

Of course, I could completely skip this particular problem by doing an openlog() myself, but I guess that I will encounter the same type of problem with some others functions.

I don't really understand why openlog_internal does not use dynamic symbol relocation to call __connect(), and if it's even possible to hook this __connect() call by using simple LD_PRELOAD mechanism.

The others way I see how it could be done:

• Load libc.so from an LD_PRELOAD with dlopen, get the address of the libc's __connect with dlsym() and then patch the function (ASM wise) to get the hook working. It seems really overkill and error prone.
• Use a modified custom libc for PHP to fix those problems directly at the source (open / connect / mmap functions...)
• Code a LKM, to redirect file access where I want. Pros : no need of ioctl(SENDFD) and no daemon outside the chroot.

I would really appreciate to learn, if it is ever possible, how I could still hook the call to __connect() issued by openlog_internal, suggestions, or links to kernel documentation related to syscall hooking and redirection.

My google searches related to "hook syscalls" found lot of references to LSM, but it seems to only allow ACLs answering "yes" or "no", but no redirection of open() paths.

Thanks for reading.

Answer 1

It's definitely not possible with LD_PRELOAD without building your own heavily-modified libc, in which case you might as well just put the redirection hacks directly inside. There are not necessarily calls to open, connect, etc. whatsoever. Instead there may be calls to a similar hidden function bound at library-creation time (not dynamically rebindable) or even inline syscalls, and this can of course change unpredictably with the version.

Your options are either a kernel module, or perhaps using ptrace on everything inside the "chroot" and modifying the arguments to syscalls whenever the tracing process encounters one that needs patching up. Neither sounds easy...

Or you could just accept that you need a minimal set of critical device nodes and files to exist inside a chroot for it to work. Using a different libc in place of glibc, if possible, would help you minimize the number of additional files needed.