infinite abort() in a backrace of a c++ program core dump

Question

I have a strange problem that I can't solve. Please help!

The program is a multithreaded c++ application that runs on ARM Linux machine. Recently I began testing it for the long runs and sometimes it crashes after 1-2 days like so:

*** glibc detected ** /root/client/my_program: free(): invalid pointer: 0x002a9408 ***

When I open core dump I see that the main thread it seems has a corrupt stack: all I can see is infinite abort() calls.

GNU gdb (GDB) 7.3 
...
This GDB was configured as "--host=i686 --target=arm-linux".
[New LWP 706]
[New LWP 700]
[New LWP 702]
[New LWP 703]
[New LWP 704]
[New LWP 705]
Core was generated by `/root/client/my_program'.
Program terminated with signal 6, Aborted.
#0  0x001c44d4 in raise ()
(gdb) bt
#0  0x001c44d4 in raise ()
#1  0x001c47e0 in abort ()
#2  0x001c47e0 in abort ()
#3  0x001c47e0 in abort ()
#4  0x001c47e0 in abort ()
#5  0x001c47e0 in abort ()
#6  0x001c47e0 in abort ()
#7  0x001c47e0 in abort ()
#8  0x001c47e0 in abort ()
#9  0x001c47e0 in abort ()
#10 0x001c47e0 in abort ()
#11 0x001c47e0 in abort ()

And it goes on and on. I tried to get to the bottom of it by moving up the stack: frame 3000 or even more, but eventually core dump runs out of frames and I still can't see why this has happened.

When I examine the other threads everything seems normal there.

(gdb) info threads
  Id   Target Id         Frame 
  6    LWP 705           0x00132f04 in nanosleep ()
  5    LWP 704           0x001e7a70 in select ()
  4    LWP 703           0x00132f04 in nanosleep ()
  3    LWP 702           0x00132318 in sem_wait ()
  2    LWP 700           0x00132f04 in nanosleep ()
* 1    LWP 706           0x001c44d4 in raise ()
(gdb) thread 5
[Switching to thread 5 (LWP 704)]
#0  0x001e7a70 in select ()
(gdb) bt
#0  0x001e7a70 in select ()
#1  0x00057ad4 in CSerialPort::read (this=0xbea7d98c, string_buffer=..., delimiter=..., timeout_ms=1000) at CSerialPort.cpp:202
#2  0x00070de4 in CScanner::readResponse (this=0xbea7d4cc, resp_recv=..., timeout=1000, delim=...) at PidScanner.cpp:657
#3  0x00071198 in CScanner::sendExpect (this=0xbea7d4cc, cmd=..., exp_str=..., rcv_str=..., timeout=1000) at PidScanner.cpp:604
#4  0x00071d48 in CScanner::pollPid (this=0xbea7d4cc, mode=1, pid=12, pid_str=...) at PidScanner.cpp:525
#5  0x00072ce0 in CScanner::poll1 (this=0xbea7d4cc) 
#6  0x00074c78 in CScanner::Poll (this=0xbea7d4cc) 
#7  0x00089edc in CThread5::Thread5Poll (this=0xbea7d360) 
#8  0x0008c140 in CThread5::run (this=0xbea7d360) 
#9  0x00088698 in CThread::threadFunc (p=0xbea7d360) 
#10 0x0012e6a0 in start_thread ()
#11 0x001e90e8 in clone ()
#12 0x001e90e8 in clone ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

(Classes and functions names are a bit wierd because I changed them -:) So, thread #1 is where the stack is corrupt, backtrace of every other (2-6) shows

Backtrace stopped: previous frame identical to this frame (corrupt stack?).

It happends because threads 2-6 are created in the thread #1.

The thing is that I can't run the program in gdb because it runs on an embedded system. I can't use remote gdb server. The only option is examining core dumps that occur not very often.

Could you please suggest something that could move me forward with this? (Maybe something else I can extract from the core dump or maybe somehow to make some hooks in the code to catch abort() call).

UPDATE: Basile Starynkevitch suggested to use Valgrind, but turns out it's ported only for ARMv7. I have ARM 926 which is ARMv5, so this won't work for me. There are some efforts to compile valgrind for ARMv5 though: Valgrind cross compilation for ARMv5tel, valgrind on the ARM9

UPDATE 2: Couldn't make Electric Fence work with my program. The program uses C++ and pthreads. The version of Efence I got, 2.1.13 crashed in a arbitrary place after I start a thread and try to do something more or less complicated (for example to put a value into an STL vector). I saw people mentioning some patches for Efence on the web but didn't have time to try them. I tried this on my Linux PC, not on the ARM, and other tools like valgrind or Dmalloc don't report any problems with the code. So, everyone using version 2.1.13 of efence be prepared to have problems with pthreads (or maybe pthread + C++ + STL, don't know).

Answer 1

My guess for the "infinite' aborts is that either abort() causes a loop (e.g. abort -> signal handler -> abort -> ...) or that gdb can't correctly interpret the frames on the stack.

In either case I would suggest manually checking out the stack of the problematic thread. If abort causes a loop, you should see a pattern or at least the return address of abort repeating every so often. Perhaps you can then more easily find the root of the problem by manually skipping large parts of the (repeating) stack.

Otherwise, you should find that there is no repeating pattern and hopefully the return address of the failing function somewhere on the stack. In the worst case such addresses are overwritten due to a buffer overflow or such, but perhaps then you can still get lucky and recognise what it is overwritten with.

Answer 2

One possibility here is that something in that thread has very, very badly smashed the stack by vastly overwriting an on-stack data structure, destroying all the needed data on the stack in the process. That makes postmortem debugging very unpleasant.

If you can reproduce the problem at will, the right thing to do is to run the thread under gdb and watch what is going on precisely at the moment when the the stack gets nuked. This may, in turn, require some sort of careful search to determine where exactly the error is happening.

If you cannot reproduce the problem at will, the best I can suggest is very carefully looking for clues in the thread local storage for that thread to see if it hints at where the thread was executing before death hit.