C - Program fails to get file descriptor only when running with GDB

Question

I'm not an expert C programmer. I'm having trouble debugging a program using GDB. (The bug I am trying to fix is unrelated to the problem I am asking about here.) My problem is that the program runs fine when I run the binary directly from a shell, but the program crashes when I run it using GDB.

Here is some information about the program which may be useful: it is a 20+ year old piece of database software, originally written for Solaris (I think) but since ported to Linux, which is setuid (but not to root, thank god).

The program crashes in GDB when trying to open a file for writing. Using GDB, I was able to determine that crash occurs because the following system call fails:

fd = open(path, O_WRONLY|O_CREAT|O_TRUNC, 0644);

For clarification: path is the path to a lockfile which should not exist. If the lock file exists, then the program shuts down cleanly before it even reaches this system call.

I do not understand why this system call would fail, since 1) The user this program runs as has rwx permissions on the directory containing path (I have verified this by examining the value of the variable stored in path), and 2) the program successfully opens the file for writing when I am not using GDB to debug it.

Are there any reasons why I cannot

Answer 1

The key turns out to be this bit:

... is setuid (but not to root, thank god).

When you run a program under (any) debugger (using any of the stop-and-inspect/modify program facilities), the kernel disables setuid-ness, even for non-root setuid.

If you think about this a bit it makes sense. Consider a game that keeps a "high scores" file, and uses "setuid games" to do this, with:

fd = open(GAME_SCORE_FILE, open_mode, file_mode);
score_data = read_scores(fd);
/* set breakpoint here or so */
if (check_for_new_high_score(current_score, score_data)) {
    printf("congratulations, you've entered the High Scores records!\n");
    save_scores(fd, score_data);
}
close(fd);

Access to the "high scores" file is protected by file permissions: only the "games" user can write to it.

If you run the game under a debugger, though, you can set a breakpoint at the marked line, and set the current_score data to some super-high value and then resume the program.

To avoid allowing debuggers to corrupt the internal data of setuid programs, the kernel simply disables setuid-ness when running code with debug facilities enabled. If you can su (or sudo or whatever) to the user, indicating that you have permission regardless of any debugging, you can then run gdb itself as that user, so that the program runs as the user it "would have" setuid-ed to.