Segmentation fault from hook on open()

Question

I am trying to create a hook on the system function open(). I've done this along the following lines.

I created a wrapper library with the following:

extern int mocked_open(const char* fn, int flags, va_list args);

int open(const char* fn, int flags, ...)
{
    int r = -1;
    va_list args;

    va_start(args, flags);
    r = mocked_open(fn, flags, args);
    va_end(args);

    return r;
}

I compile this into libwrapper.so, which I load using LD_PRELOAD.

The implementation of mocked_open() is as follows (I use the CPPUtest framework):

int mocked_open(const char* fn, int flags, va_list args)
{
    if (strncmp(fn, test_device_id, 11) == 0)
    {
        return mock().actualCall("open").returnValue().getIntValue();
    }
    else
    {
        int r = -1;
        int (*my_open)(const char*, int, ...);
        void* fptr = dlsym(RTLD_NEXT, "open");
        memcpy(&my_open, &fptr, sizeof(my_open));

        if (flags & O_CREAT)
        {
            r = my_open(fn, flags, va_arg(args, mode_t));
        }
        else
        {
            r = my_open(fn, flags);
        }

        return r;
    }
}

The test_device_id is a simple string ("test_device"), which I hope is not used elsewhere.

During running the tests the executable crashes with a segmentation fault. I've traced this down to the GCC profiling functionality, which wants to open/create a bunch of .gcda files and calls open() for this.

After some debugging with strace (per suggestion below), I found that the line r = my_open(fn, flags, va_arg(args, mode_t)); is indeed the culprit. It is being called recursively, or so it seems: I see a lot of calls to this line, without the function returning. Then a segfault. The file being opened is the corresponding .gcda file (for profiling). In fact, the segfault only occurs with profiling enabled.

Answer 1

Try this

typedef int (*OpenFunction)(const char* fn, int flags, ...);

then

OpenFunction function;
void      **pointer;

pointer  = (void **)&function;
*pointer = dlsym(RTLD_NEXT, "open");

this is a complete working example

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <dlfcn.h>
#include <unistd.h>
#include <fcntl.h>

#include <errno.h>

typedef int (*OpenFunction)(const char* fn, int flags, ...);

int main(int argc, char **argv)
{
    OpenFunction function;
    void       *dl;
    int         fd;
    void      **pointer;

    if (argc < 2)
        return -1;
    pointer  = (void **)&function;
    *pointer = dlsym(RTLD_NEXT, "open");

    fd = function(argv[1], O_RDONLY);
    if (fd != -1)
    {
        printf("file opened succesfully\n");
        close(fd);
    }
    else
    {
        printf("%s: cannot open the file\n", strerror(errno));
    }
    return 0;
}

Answer 2

When you compile with gcov profiling enabled, the compiler inserts extra code into your functions, to keep track of which code has been executed. In rough pseudocode, that inserted code will do (among other things):

if (!output_file_has_been_opened) {
    fd = open(output_filename, ...);
    check_ok(fd);
    output_file_has_been_opened = TRUE;
    track_coverage();
}

... so if the output file hasn't yet been successfully opened (as at the start of your program), it will attempt to open it. Unfortunately, in this case that will call your mocked open() function - which has the same inserted code; since the file still hasn't been successfully opened, and since the gcov code isn't aware that there's something unusual going on, it will attempt the open() call again - and this is what's causing the recursion (and eventual segfault, once the stack is exhausted).