Debugging techniques without debugging tools

Question

I find myself in the difficult situation of having to debug a Qt application without almost any debugging tool: the application seems to start using more and more CPU as it is running the same action again and again; after many hours CPU is completely saturated.

The application runs on a ARM Linux embedded device where gdb seems not to work, maybe hard-to-discover problems with the provided toolchain. strace seems only to report timer activities (this is an OpenGL application so this is expected). ltrace is not available and compiling it resulted in a difficult task, maybe useless. I didn't write the application but the source code is available.

Is there anything else I can do to discover what the application is busy doing when consuming that much resources? Any way I have to trace all the method calls the application does? Is there any other technique I can use to try to guess the problem or where to focus my attention?

EDIT: This is one of the problems with gdb: Only question marks in backtrace reported by gdb on ARM. Even writing a ten lines application simulating a segfault results in this.

Answer 1

Can you enable core dumps on the machine? Then when it is playing up, you can send it a SIGABRT and copy the core dump to your dev machine, and examine it with a cross-debugger, with source and unstripped executable available.

It's also important to learn the bitter lesson for next time, don't use such a badly supported toolchain.

If it's an option, you could try another toolchain with at least gdbserver if not gdb support. I have been quite happy with the CodeSourcery ARM Lite toolchain.

EDIT: gdb for your situation comes in two flavours:

• a cross-gdb that runs on your development host
• a native gdb that runs on your target

gdbserver allows you to run your cross-gdb on your development host and connect to the target to remotely debug something running on it. So a core dump or gdbserver are two ways to use a cross-gdb to inspect something on the target, but gdbserver alone won't help you much.

If your cross-compiler is something like arm-none-linux-gnueabi-gcc, see if you have an arm-none-linux-gnueabi-gdb available on your dev host.

Answer 2

You can try to place some debugging code in your application.

Choose some signal, like SIGINT. Add signal handler for this signal. In this handler print the stack trace or at least the instruction pointer value. Then start application and send SIGINT several times to see what your application is doing.