Post-mortem debugging of a Mono AOT-compiled program

Question

I have a large program, written in C# and running on Linux systems using Mono, that occasionally crashes and causes the mono.bin process to dump core.

I ran gdb on some of the core dump files, but it wasn't very useful because the backtraces don't have the names of the C# functions in them. According to this discussion I found:

It won't work. The information required to construct managed stack traces is contained in runtime data structures, and it is only available while the program is running. You can AOT your application, then you will have more usable stack traces.

So, I did. I AOT-compiled all of my C# DLL and EXE files. Using the --aot=write-symbols option. For a test version of my program that crashes on purpose so I could check if this makes the backtraces more useful. And so far, it hasn't. The backtrace from the main thread looks like:

#0  0xb7fc8402 in __kernel_vsyscall ()
#1  0x00556df0 in raise () from /lib/libc.so.6
#2  0x00558701 in abort () from /lib/libc.so.6
#3  0x080e59b5 in ?? ()

Another thread has:

#0  0xb7fc8402 in __kernel_vsyscall ()
#1  0x005f6753 in poll () from /lib/libc.so.6
#2  0xb6f735a7 in Mono_Unix_UnixSignal_WaitAny ()
   from /opt/novell/mono/lib/libMonoPosixHelper.so
#3  0xb5416578 in ?? ()

And other threads seem to have been idle (in nanosleep, pthread_cond_timedwait, pthread_cond_wait, sem_timedwait, or sem_wait). But the thing all backtraces have in common is that they end with that annoying in ?? (), and never list any function names from "my" code.

I think this is related to some messages that gdb printed when it started up; for example,

Reading symbols from /xyz/mono/log4net.dll.so...(no debugging symbols found)...done.
Loaded symbols for /xyz/mono/log4net.dll.so
Reading symbols from /xyz/mono/Contoso.Util.dll.so...(no debugging symbols found)...done.
Loaded symbols for /xyz/mono/Contoso.Util.dll.so
Reading symbols from /xyz/mono/Contoso.Printing.dll.so...(no debugging symbols found)...done.
Loaded symbols for /xyz/mono/Contoso.Printing.dll.so
Reading symbols from /xyz/mono/Contoso.LegacyDataConverter.dll.so...(no debugging symbols found)...done.
Loaded symbols for /xyz/mono/Contoso.LegacyDataConverter.dll.so

Why do all the *.dll.so files have "no debugging symbols found"? Do the DLLs themselves need to be built in "debug" mode or something?

And more generally, is there a way to get the managed stack trace from a Mono core dump? (Without using mono_pmip, because that's only available when the process is running.)

Answer 1

Would it be possible to set suspend-on-sigserv then attach when the process crashes? I'm asssuming this is a live environment so may not be possible.

If you can do that you should be able to find the information you are after.

From the docs:

MONO_DEBUG

If set, enables some features of the runtime useful for debugging. This variable should contain a comma separated list of debugging options. Currently, the following options are supported:

...

suspend-on-sigsegv

This option will suspend the program when a native SIGSEGV is received. This is useful for debugging crashes which do not happen under gdb, since a live process contains more information than a core file.

Answer 2

I have no idea about mono.. But from looking at stacktrace provided.. it show not able to get Symbols for mono-runtime. here is link which gives debugging with gdb. You need mono-runtime with symbols i.e so library with symbols...So you need to install mono-runtime-dbg...You can use tools like apt-get, yum, wget to install it.

I have unbuntu installed.. which shows following output....

$ apt-cache search mono-runtime-dbg
mono-runtime-dbg - Mono runtime, debugging symbols

Then
$ apt-get install mono-runtime-dbg

Hope you find it helpful...