How do you debug the bug that only appears when the load is huge?

Question

We are currently developing a cluster manager software in C. If several nodes connect to the manager, it works perfect, but if we use some tools to simulate 1000 nodes to connect the manager, it will sometimes work in unexpected ways.

How can one debug this kind of bug? It only appears when the load(connection/nodes) is large?

If I use gdb to debug step by step, the app never malfunctions.

Answer 1

How to debug this kind of bug?

In general, you want to use at least these techniques:

1. Make sure the code compiles and links without warnings. The -Wall is a good start, but -Wextra is better.
2. Make sure the application has designed-in logging and tracing, which can be turned on or off, and which has sufficient details to debug these kinds of issues, and low overhead.
3. Make sure the code has good unit-test coverage.
4. Make sure the tests are sanitizer-clean.

there's also no warning in valgrind check.

It's not clear whether you've simply ran the target application under Valgrind, or whether you also have the unit tests, and the tests are Valgrind-clean. It's also not clear whether you've observed the application mis-behavior under Valgrind or not.

Valgrind used to be the best tool available for heap and unintialized memory problems, but in 2017 this is no longer the case.

Compiler-based Address, Thread and Memory sanitizers catch significantly wider class of errors (e.g. global and stack overflows, and data races), and you should run your unit tests under all of them.

When all of the above still fails to find the problem, you may be able to run the real application instrumented with sanitizers.

Lastly, there are tools like GDB tracing and systemtap -- they are harder to learn, but give you significant power. Overview here.

Answer 2

Sadly the debugger is less useful for debugging concurrency/load issues.

Keep adding logs/printfs, trigger the issue with load testing, then try to narrow it down with more logs/printfs. Repeat.

The faster it is to trigger the bug the faster this will converge. Also prefer the classic "bisection" / "binary search" technique when adding logs - try to narrow down the areas you're looking at by at least half every time.