Best way to print information when debugging a race condition

Question

I am debugging an application to fix a segmentation fault that I suspect to be caused by a race condition.

I'd like to put some print statements in the code, but I know for experience that adding calls to printf is not recommended since this could change the behavior of the threads and in some case hide the bug.

Looking at other options, I have seen that with gdb it is possible to use break points to print something and then automatically continue the execution:

break foo
commands
silent
printf "Called foo: x is %d\n",x
cont
end

Is this any better then putting a printf in my code?

I know that gdb has a also Tracepoints but they only work with gdbserver and this is an additional level of complication that I would prefer to avoid at the moment.

Additional information: the application is written in C and it runs on Linux.

Answer 1

Is this any better then putting a printf in my code?

No, it's much worse. Every breakpoint that is hit in GDB triggers the following chain of events:

• context switch from running thread to GDB
• GDB stops all other threads (assuming default all-stop mode)
• GDB evaluates breakpoint commands
• GDB resumes all threads (this is itself a complicated multi-step process, which I would not go into here).

This is at least an order of magnitude more expensive and disruptive than a simple printf call, and is very likely to hide whatever race you were trying to debug.

The bottom line is that GDB is in general completely unsuitable for debugging data races.

I second the ThreadSanitizer recommendation by Christopher Ian Stern.

The only problem with this bug is that I am doing the debug on a production machine where I cannot install other SW.

First, ThreadSanitizer instruments your existing program. It has a runtime library, but that could be statically linked into your binary. There is nothing that you need to install on your production machine.

Second, ThreadSanitizer detects data races even when they do not cause visible behavior changes. It may well turn out that you don't need to run on your production machine at all: simply running your tests (you do have tests, right?) on your development machine may prove to be sufficient.

Answer 2

Since you are on Linux, I would recomend ThreadSanitizer. That is using a recent version of gcc or clang and passing -fsanitize=thread to the build. This isn't a printf repacment but should tell you explicitly about any race conditions in your code. Even after you solve this problem if you are working with multithreaded code, you will want to have this tool available. Alternately, or in addition, I have had good results with Valgrind's http://valgrind.org Data Race Detector, but I would start with ThreadSanitizer.