Debugging a Ruby segfault

Question

How do I determine if a segfault is due to inconsistent libraries, or a bug in some gem I'm using?

$ uname -a
Linux [redacted] 3.2.0-24-generic #39-Ubuntu SMP Mon May 21 16:52:17 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
$ ruby1.9.1 --version
ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux]

$ CPUPROFILE_OBJECTS=1 CPUPROFILE=/tmp/my_app_profile_objects RUBYOPT="-r`gem1.9.1 which perftools | tail -1`" ruby1.9.1 -e '[].map'
-e:1: [BUG] Segmentation fault
ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0004 p:---- s:0009 b:0009 l:000008 d:000008 CFUNC  :map
c:0003 p:0010 s:0006 b:0006 l:0011f8 d:0002f8 EVAL   -e:1
c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH
c:0001 p:0000 s:0002 b:0002 l:0011f8 d:0011f8 TOP   

-- Ruby level backtrace information ----------------------------------------
-e:1:in `<main>'
-e:1:in `map'

-- C level backtrace information -------------------------------------------
/usr/lib/libruby-1.9.1.so.1.9(+0x155b29) [0x7fd39f1dab29] ../sysdeps/x86_64/multiarch/strcpy-ssse3.S:1638
/usr/lib/libruby-1.9.1.so.1.9(+0x57709) [0x7fd39f0dc709] vfscanf.c:1777
/usr/lib/libruby-1.9.1.so.1.9(rb_bug+0xb7) [0x7fd39f0dd137] vfscanf.c:1796
/usr/lib/libruby-1.9.1.so.1.9(+0xf604f) [0x7fd39f17b04f] wcfuncs.c:49
/lib/x86_64-linux-gnu/libc.so.6(+0x364c0) [0x7fd39ecfe4c0] ../sysdeps/posix/killpg.c:38
/usr/lib/libruby-1.9.1.so.1.9(rb_check_type+0xd8) [0x7fd39f0de7e8] vfscanf.c:1963
/usr/lib/libruby-1.9.1.so.1.9(rb_data_typed_object_alloc+0xad) [0x7fd39f0f340d] iofdopen.c:51
/usr/lib/libruby-1.9.1.so.1.9(+0x54f9f) [0x7fd39f0d9f9f] vfscanf.c:2014
/usr/lib/libruby-1.9.1.so.1.9(rb_enumeratorize+0x33) [0x7fd39f0dae33] vfscanf.c:2373
/usr/lib/libruby-1.9.1.so.1.9(+0x14e9a1) [0x7fd39f1d39a1] ../sysdeps/x86_64/multiarch/../strcmp.S:293
/usr/lib/libruby-1.9.1.so.1.9(+0x145912) [0x7fd39f1ca912] ../sysdeps/x86_64/multiarch/memcpy-ssse3.S:2001
/usr/lib/libruby-1.9.1.so.1.9(+0x14b31d) [0x7fd39f1d031d] ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S:2032
/usr/lib/libruby-1.9.1.so.1.9(rb_iseq_eval_main+0xaf) [0x7fd39f1d769f] ../sysdeps/x86_64/multiarch/../strcmp.S:1811
/usr/lib/libruby-1.9.1.so.1.9(+0x5b172) [0x7fd39f0e0172] vfscanf.c:1471
/usr/lib/libruby-1.9.1.so.1.9(ruby_exec_node+0x1d) [0x7fd39f0e0c5d] vfscanf.c:2288
/usr/lib/libruby-1.9.1.so.1.9(ruby_run_node+0x1e) [0x7fd39f0e280e] psiginfo.c:209
ruby1.9.1() [0x4007db]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7fd39ece976d] libc-start.c:226
ruby1.9.1() [0x400809]

-- Other runtime information -----------------------------------------------

* Loaded script: -e

* Loaded features:

    0 enumerator.so
    1 /usr/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
    2 /usr/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
    3 /usr/lib/ruby/1.9.1/rubygems/defaults.rb
    4 /usr/lib/ruby/1.9.1/x86_64-linux/rbconfig.rb
    5 /usr/lib/ruby/1.9.1/rubygems/deprecate.rb
    6 /usr/lib/ruby/1.9.1/rubygems/exceptions.rb
    7 /usr/lib/ruby/1.9.1/rubygems/custom_require.rb
    8 /usr/lib/ruby/1.9.1/rubygems.rb
    9 /var/lib/gems/1.9.1/gems/perftools.rb-2.0.0/lib/perftools.so

* Process memory map:

00400000-00401000 r-xp 00000000 08:07 704429                             /usr/bin/ruby1.9.1
00600000-00601000 r--p 00000000 08:07 704429                             /usr/bin/ruby1.9.1
00601000-00602000 rw-p 00001000 08:07 704429                             /usr/bin/ruby1.9.1
01464000-016e4000 rw-p 00000000 00:00 0                                  [heap]
7fd39c3e1000-7fd39cf53000 rw-p 00000000 00:00 0 
7fd39cf53000-7fd39cf68000 r-xp 00000000 08:06 5443                       /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd39cf68000-7fd39d167000 ---p 00015000 08:06 5443                       /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd39d167000-7fd39d168000 r--p 00014000 08:06 5443                       /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd39d168000-7fd39d169000 rw-p 00015000 08:06 5443                       /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd39d169000-7fd39d24b000 r-xp 00000000 08:07 137087                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7fd39d24b000-7fd39d44a000 ---p 000e2000 08:07 137087                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7fd39d44a000-7fd39d452000 r--p 000e1000 08:07 137087                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7fd39d452000-7fd39d454000 rw-p 000e9000 08:07 137087                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7fd39d454000-7fd39d469000 rw-p 00000000 00:00 0 
7fd39d469000-7fd39d47d000 r-xp 00000000 08:06 144897                     /var/lib/gems/1.9.1/gems/perftools.rb-2.0.0/lib/perftools.so
7fd39d47d000-7fd39d67c000 ---p 00014000 08:06 144897                     /var/lib/gems/1.9.1/gems/perftools.rb-2.0.0/lib/perftools.so
7fd39d67c000-7fd39d67d000 r--p 00013000 08:06 144897                     /var/lib/gems/1.9.1/gems/perftools.rb-2.0.0/lib/perftools.so
7fd39d67d000-7fd39d67e000 rw-p 00014000 08:06 144897                     /var/lib/gems/1.9.1/gems/perftools.rb-2.0.0/lib/perftools.so
7fd39d67e000-7fd39d682000 rw-p 00000000 00:00 0 
7fd39d682000-7fd39d684000 r-xp 00000000 08:07 157727                     /usr/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
7fd39d684000-7fd39d884000 ---p 00002000 08:07 157727                     /usr/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
7fd39d884000-7fd39d885000 r--p 00002000 08:07 157727                     /usr/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
7fd39d885000-7fd39d886000 rw-p 00003000 08:07 157727                     /usr/lib/ruby/1.9.1/x86_64-linux/enc/trans/transdb.so
7fd39d886000-7fd39d888000 r-xp 00000000 08:07 157743                     /usr/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
7fd39d888000-7fd39da87000 ---p 00002000 08:07 157743                     /usr/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
7fd39da87000-7fd39da88000 r--p 00001000 08:07 157743                     /usr/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
7fd39da88000-7fd39da89000 rw-p 00002000 08:07 157743                     /usr/lib/ruby/1.9.1/x86_64-linux/enc/encdb.so
7fd39da89000-7fd39e16c000 r--p 00000000 08:07 134744                     /usr/lib/locale/locale-archive
7fd39e16c000-7fd39e265000 r-xp 00000000 08:06 5454                       /lib/x86_64-linux-gnu/libm-2.15.so
7fd39e265000-7fd39e464000 ---p 000f9000 08:06 5454                       /lib/x86_64-linux-gnu/libm-2.15.so
7fd39e464000-7fd39e465000 r--p 000f8000 08:06 5454                       /lib/x86_64-linux-gnu/libm-2.15.so
7fd39e465000-7fd39e466000 rw-p 000f9000 08:06 5454                       /lib/x86_64-linux-gnu/libm-2.15.so
7fd39e466000-7fd39e46f000 r-xp 00000000 08:06 5430                       /lib/x86_64-linux-gnu/libcrypt-2.15.so
7fd39e46f000-7fd39e66f000 ---p 00009000 08:06 5430                       /lib/x86_64-linux-gnu/libcrypt-2.15.so
7fd39e66f000-7fd39e670000 r--p 00009000 08:06 5430                       /lib/x86_64-linux-gnu/libcrypt-2.15.so
7fd39e670000-7fd39e671000 rw-p 0000a000 08:06 5430                       /lib/x86_64-linux-gnu/libcrypt-2.15.so
7fd39e671000-7fd39e69f000 rw-p 00000000 00:00 0 
7fd39e69f000-7fd39e6a1000 r-xp 00000000 08:06 5435                       /lib/x86_64-linux-gnu/libdl-2.15.so
7fd39e6a1000-7fd39e8a1000 ---p 00002000 08:06 5435                       /lib/x86_64-linux-gnu/libdl-2.15.so
7fd39e8a1000-7fd39e8a2000 r--p 00002000 08:06 5435                       /lib/x86_64-linux-gnu/libdl-2.15.so
7fd39e8a2000-7fd39e8a3000 rw-p 00003000 08:06 5435                       /lib/x86_64-linux-gnu/libdl-2.15.so
7fd39e8a3000-7fd39e8aa000 r-xp 00000000 08:06 5508                       /lib/x86_64-linux-gnu/librt-2.15.so
7fd39e8aa000-7fd39eaa9000 ---p 00007000 08:06 5508                       /lib/x86_64-linux-gnu/librt-2.15.so
7fd39eaa9000-7fd39eaaa000 r--p 00006000 08:06 5508                       /lib/x86_64-linux-gnu/librt-2.15.so
7fd39eaaa000-7fd39eaab000 rw-p 00007000 08:06 5508                       /lib/x86_64-linux-gnu/librt-2.15.so
7fd39eaab000-7fd39eac3000 r-xp 00000000 08:06 5502                       /lib/x86_64-linux-gnu/libpthread-2.15.so
7fd39eac3000-7fd39ecc2000 ---p 00018000 08:06 5502                       /lib/x86_64-linux-gnu/libpthread-2.15.so
7fd39ecc2000-7fd39ecc3000 r--p 00017000 08:06 5502                       /lib/x86_64-linux-gnu/libpthread-2.15.so
7fd39ecc3000-7fd39ecc4000 rw-p 00018000 08:06 5502                       /lib/x86_64-linux-gnu/libpthread-2.15.so
7fd39ecc4000-7fd39ecc8000 rw-p 00000000 00:00 0 
7fd39ecc8000-7fd39ee7b000 r-xp 00000000 08:06 5422                       /lib/x86_64-linux-gnu/libc-2.15.so
7fd39ee7b000-7fd39f07a000 ---p 001b3000 08:06 5422                       /lib/x86_64-linux-gnu/libc-2.15.so
7fd39f07a000-7fd39f07e000 r--p 001b2000 08:06 5422                       /lib/x86_64-linux-gnu/libc-2.15.so
7fd39f07e000-7fd39f080000 rw-p 001b6000 08:06 5422                       /lib/x86_64-linux-gnu/libc-2.15.so
7fd39f080000-7fd39f085000 rw-p 00000000 00:00 0 
7fd39f085000-7fd39f0f0000 r-xp 00000000 08:07 158047                     /usr/lib/libruby-1.9.1.so.1.9.1
7fd39f0f0000-7fd39f0f3000 rwxp 0006b000 08:07 158047                     /usr/lib/libruby-1.9.1.so.1.9.1
7fd39f0f3000-7fd39f275000 r-xp 0006e000 08:07 158047                     /usr/lib/libruby-1.9.1.so.1.9.1
7fd39f275000-7fd39f474000 ---p 001f0000 08:07 158047                     /usr/lib/libruby-1.9.1.so.1.9.1
7fd39f474000-7fd39f479000 r--p 001ef000 08:07 158047                     /usr/lib/libruby-1.9.1.so.1.9.1
7fd39f479000-7fd39f47d000 rw-p 001f4000 08:07 158047                     /usr/lib/libruby-1.9.1.so.1.9.1
7fd39f47d000-7fd39f499000 rw-p 00000000 00:00 0 
7fd39f499000-7fd39f4bb000 r-xp 00000000 08:06 5402                       /lib/x86_64-linux-gnu/ld-2.15.so
7fd39f59b000-7fd39f6a1000 rw-p 00000000 00:00 0 
7fd39f6b4000-7fd39f6b5000 rw-p 00000000 00:00 0 
7fd39f6b5000-7fd39f6b6000 ---p 00000000 00:00 0 
7fd39f6b6000-7fd39f6bb000 rw-p 00000000 00:00 0 
7fd39f6bb000-7fd39f6bc000 r--p 00022000 08:06 5402                       /lib/x86_64-linux-gnu/ld-2.15.so
7fd39f6bc000-7fd39f6be000 rw-p 00023000 08:06 5402                       /lib/x86_64-linux-gnu/ld-2.15.so
7fff507f8000-7fff50819000 rw-p 00000000 00:00 0                          [stack]
7fff50941000-7fff50942000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]


[NOTE]
You may have encountered a bug in the Ruby interpreter or extension libraries.
Bug reports are welcome.
For details: http://www.ruby-lang.org/bugreport.html

Aborted (core dumped)

Answer 1

You can find out which part of the software caused a segmentation fault by - and this is probably not a surprise to you - debugging!. As you can see...

Aborted (core dumped)

...there is a core dump.

Just fire up gdb on your executable and the core dump and find out what the parameters to that strcpy (see the first line of your backtrace) were. Then walk backwards trough the stack traces and the code. It seems like debugging symbols have been compiled in, so you are lucky and you can see which function is actually called and what parameters are supplied - at least if the bug did not corrupt the stack.

However, If you are not too familiar with debugging C or assembly code, you probably want to wait for the developers of perftools to debug this issue.