Use Apport crash dump to debug Python program

Question

I have a Python program that sometimes crashes due to a "double free or corruption" error. I am trying to figure out where this is happening (possibly in one of the many libraries I am using) so that I can prevent it from crashing. To that end I enabled core dumps, and I now have an Apport .crash file to work with.

Here's where I'm stuck. How do I load the core dump into gdb or something else that will let me see whatever stacktrace information is available?

apport-retrace seems like it would be great, but won't load because there's no package in the .crash file:

ERROR: report file does not contain one of the required fields: CoreDump DistroRelease Package ExecutablePath

I also can't figure out how to load it directly into gdb. I've tried gdb /usr/bin/python <crashfile> on the full .crash file, on just the "CoreDump" portion of the .crash file, and on a base64-decoded version of the "CoreDump" section. Each time I've gotten this error:

<crashfile> is not a core dump: File format not recognized

Is there a way I can either use apport-retrace without needing a package or pull the core dump out of the crash file in a way that gdb can use it?

Answer 1

It turns out it was fairly simple to modify the .crash file to allow apport-retrace to open it. I simply needed to add

Package: python2.7

to the file. For good measure, I also made sure that the "ExecutablePath" was for Python:

ExecutablePath: /usr/bin/python2.7

In my case, the executable path was previously a different file (one specific to my program). I don't know if this step was actually necessary.

After doing this, I could run apport-retrace -g <crashfile> to open it up in gdb and then use bt to extract the stacktrace.

Answer 2

1) Instead of modifying the .crash file, you can add the -R switch to apport-retrace. This generates a Packages: field, which particularly may be missing if there's a crash in package system itself.

Also, if you merely want to output the backtrace, you can use the -s switch in place of -g. -g loads the interactive debugger. Together this may look like:

apport-retrace -C /tmp/apportcache/ -R -s -S system \
    /var/crash/$executable-path.$uid.crash > output.trace

2) An alternative is to disable apport by setting a kernel parameter and so produce a binary core file for use with gdb -c.

ulimit -c unlimited
sysctl kernel.core_pattern=/var/crash/core.%e.%p

3) See also https://wiki.ubuntu.com/DebuggingProgramCrash for a GUI method once you have apport-retrace installed.