How can I replay a multithreaded application?

Question

I want to record synchronization operations, such as locks, sempahores, barriers of a multithreaded application, so that I can replay the recorded application later on, for the purpose of debugging.

On way is to supply your own lock, sempaphore, condition variables etc.. functions that also do logging, but I think that is an overkill, because down under they must be using some common synchronization operations.

So my question is which synchronization operations I should log so that I require minimum modifications to my program. In other words, which are the functions or macros in glibc and system calls over which all these synchronization operations are built? So that I only modify those for logging and replaying.

Answer 1

The best I can think of is debugging with gdb in 'record' mode:

• Gdb process record/replay execution log

According to this page: GDB Process Record threading support is underway, but it might not be complete yet.

---

Less strictly answering your question, may I suggest

• Helgrind
• DRD

On other platforms, several other threading checkers exist, but I haven't got much experience with them.

Answer 2

In your case, an effective method of "logging" systems calls on Linux may be to use the LD_PRELOAD trick, and over-ride the actual system calls with your own versions of the calls that will log the use of the call and then forward to the actual system call.

A more extensive example is given here in Linux Journal.

As you can see at these links, the basic gist of the "trick" is that you can make the system load your own dynamic library before any other system libraries, such as pthreads, etc., and then mask the calls to those library functions by placing your own versions of those functions as the precendent. You can then, inside your over-riding function, log the use of the original function, as well as pass on the arguments to the actual call you're attempting to log.

The nice thing about this method is it will catch pretty much any call you can make, both a function that remains entirely in user-land, as well as a function that will make a kernel call.

Answer 3

So GDB record mode doesn't support multithreading, but the RR record/replay system absolutely does: https://rr-project.org/ .

For a commercial solution with fewer technical restrictions, there's also UDB: https://undo.io/solutions/ .

I've worked on debuggers for some years now and from what I've seen, the GDB record+replay stuff is really not ready for primetime, for this and other reasons (eg, slowdown & huge memory requirements).

If you can get it to work in your dev environment, record+replay/reversible debugging can be pretty gamechanging for your workflow; I hope you find a way to leverage it.