Debugging a running daemon using gdb

Question

I am developing a high traffic network C server application that runs as a daemon. Under some circumstances, the app crashes (always without core). How I can debug the running daemon with gdb to find the place that generates the SIGSEGV?

Explanatory notes:

1. I know how to attach using gdb to a running process using attach command

2. After attaching to the process, it stops. If I run then "continue", gdb remains blocked if the program does not crash. If I press CTRL-C, the process is exiting and I am unable to simply detach gdb.

So the question is: is there a way to continue the process without the gdb being stuck but being able to detach if the process does not crash?

Answer 1

Try async mode and "continue &":

Save below to non-stop.gdb

set target-async on
set pagination off
set non-stop on

Then run:

$ gdb -x non-top.gdb
(gdb) !pgrep YOUR-DAEMON
1234
(gdb) attach 1234
(gdb) continue -a &
(gdb)

Answer 2

This page attach/detach says that the detach command would work inside gdb.

If you want to catch a segmentation fault in an application, you will have to run the application from the debugger. Then when the signal is caught you can use where or bt to see a stack trace of the application. Of course you can not continue the application after it faulted, how should it recover? If you expect to trigger the fault soon, you can attach to the running process and again await the fault in the debugger.

If you want a stack trace after the fault occurred, then you really need a core file as there will be no process to attach to. Now if your daemon is started as part of the system it may be hard to get the configuration to dump core, plus you may not want other applications to leave core dumps all over the place. So then I'd advice to stop the system daemon and start it again in your user space, then you can allow it to dump core. If it is really essential that it starts up as part of the system, then see if the start-up of the daemon is confined to a single sub-shell and use ulimit -c in that sub-shell to set an appropriate maximum size for the core dump.