How to debug FUSE filesystem crash in Linux

Question

Currently I am developing an application using FUSE filesystem module in Linux (2.6 Kernel) in C language. Due to some programming error, the application crashes after mounting the filesystem. Since I am a novice developer in Linux/C environment. Could you please let me tell me possible options to debug such programs?

Answer 1

There are a couple features of FUSE that can make it difficult to debug: it normally runs in the background (which means it detaches from stdin/out) and is multithreaded (which can introduce race conditions and is more complicated to debug with gdb). Luckily, both features can be disabled:

1. Use the -f switch to keep your application in the foreground. This will make your printf lines work.
2. Use the -s switch to disable multithreading. Disabling multithreading will limit performance, but will also hide certain bugs (race conditions), simplify the use of gdb, and ensure printf output is readable (when multiple threads call printf at about the same time their output can get mixed up).

I'd also recommend reading Geoff Kuenning's FUSE documentation.

Answer 2

Run your fuse client with the -d option.

Answer 3

First, make sure you're compiling with debugging symbols enabled (-g option to gcc). Before you run your program, enable core dumps with the shell command:

ulimit -c unlimited

Then when the application crashes, it'll leave a core file in the current working directory (as long as it can write to it).

You can then load the core file in the gdb debugger:

gdb <executable file> <core file>

...and it'll show you where it crashed, and let you examine variables and so forth.

Answer 4

You can use Valgrind with FUSE, however read this first to learn about a setuid work-around. I actually do the following as a convenience for others who might need to debug my file system:

#include <valgrind/valgrind.h>

if (RUNNING_ON_VALGRIND) {
    fprintf(stderr,
        "******** Valgrind has been detected by %s\n"
        "******** If you have difficulties getting %s to work under"
        " Valgrind,\n"
        "******** see the following thread:\n"
        "******** http://www.nabble.com/valgrind-and-fuse-file-systems"
        "-td13112112.html\n"
        "******** Sleeping for 5 seconds so this doesn't fly by ....",
            progname, progname);
    sleep(5);
    fprintf(stderr, "\n");
}

I work on FUSE a lot .. and 90% of the time my crashes are due to a leak which causes the OOM killer to take action, dereferencing a bad pointer, double free(), etc. Valgrind is a great tool to catch that. GDB is helpful, but I've found Valgrind to be indispensable.