Tools to visualize multithreaded C++ application call graph, multithreaded code coverage?

Question

I would like to know if there are tools that can

• Help visualize call graph of a large multi-threaded application.
• Specifically I want to see how multiple threads interleaves on one core / executes simultaneously on multiple cores.
• The tool would ideally identify possible wait/deadlock/race conditions.

• Ultimately I want to do code coverage in terms of how threads interacts with each other during runtime (multi-thread-wise code coverage tool) so as to find potential multi-threaded bugs.

  I apologize if I haven't explained my question clearly and I would love to provide any details.

Answer 1

The VTune Profiler from Intel can do some of what you ask. From the VTune site:

Locks and Waits: Use the Intel® performance profiling tools to quickly find a common cause of slow performance in parallel programs: waiting too long on a lock while the cores are underutilized during the wait.

Timeline Visualizes Thread Behavior: See when threads are running and waiting, and when transitions occur.

If you were looking for something that is open source/free, then Valgrind has an experimental tool called Helgrind that supposedly finds races in multi-threaded programs. I can't comment on it, I haven't used it.

I should note that I haven't been successful in utilizing these or other profilers for multi-threaded debugging and optimizations, and instead I have developed my own techniques.

For identifying lock contention my preferred technique is to use an extended Mutex class that records all the operations done on each instance. I do this in a very lightweight way, so that the application performance doesn't change in a big way.

To identify race conditions I find the brute force approach the best. I just design a test that can be run for an extended period of time, some times this is hours, or days, depending on the case. And I always run my test on at least two different platforms (more if I can), since different OSes use different schedulers and that gives you better coverage.