How do I diagnose this PHP segmentation fault?

Question

I am running a command-line PHP job that is resulting in a Segmentation Fault. This job has worked for a long time, but it processes something that is emailed in. Apparently there's something in this email that's break it, but I have no idea what. If I put the core file in gdb, it's really no help:

$ gdb /usr/local/bin/php core.20381  GNU gdb (GDB) 7.1-ubuntu Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.  Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/local/bin/php...done. [New Thread 20381]  warning: Can't read pathname for load map: Input/output error. Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypt.so.1 Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/librt.so.1 Reading symbols from /usr/lib/libmysqlclient_r.so.15...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libmysqlclient_r.so.15 Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libpthread.so.0 Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libz.so.1 Reading symbols from /usr/lib/libmcrypt.so.4...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libmcrypt.so.4 Reading symbols from /usr/lib/libltdl.so.7...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libltdl.so.7 Reading symbols from /lib/libpng12.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libpng12.so.0 Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libnsl.so.1 Reading symbols from /usr/lib/libcurl-gnutls.so.4...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libcurl-gnutls.so.4 Reading symbols from /usr/lib/libxml2.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libxml2.so.2 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libresolv.so.2 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /usr/lib/libidn.so.11...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libidn.so.11 Reading symbols from /usr/lib/liblber-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/liblber-2.4.so.2 Reading symbols from /usr/lib/libldap_r-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libldap_r-2.4.so.2 Reading symbols from /usr/lib/libgssapi_krb5.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgssapi_krb5.so.2 Reading symbols from /usr/lib/libgnutls.so.26...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgnutls.so.26 Reading symbols from /lib/libgcrypt.so.11...(no debugging symbols found)...done. Loaded symbols for /lib/libgcrypt.so.11 Reading symbols from /usr/lib/libsasl2.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libsasl2.so.2 Reading symbols from /usr/lib/libkrb5.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libkrb5.so.3 Reading symbols from /usr/lib/libk5crypto.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libk5crypto.so.3 Reading symbols from /lib/libcom_err.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libcom_err.so.2 Reading symbols from /usr/lib/libkrb5support.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libkrb5support.so.0 Reading symbols from /lib/libkeyutils.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libkeyutils.so.1 Reading symbols from /usr/lib/libtasn1.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libtasn1.so.3 Reading symbols from /lib/libgpg-error.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libgpg-error.so.0 Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libgcc_s.so.1 Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_files.so.2 Can't read symbols from system-supplied DSO at 0x7fffb4bfc000: File truncated Core was generated by `/usr/local/bin/php /src/prod/current/utils/cron.php --run=/utils/proces'. Program terminated with signal 11, Segmentation fault. #0  match (     eptr=0x1370a887 "/td>\n\t\t\t\t\t\t\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t</table>\n\t\t\t</td>\n\t\t</tr>\n\t\t\n\t\t<tr>\n\t\t\t<td style=\"border-top:3px solid #efefef; padding:20px 10px;\">\n\t\t\t\t<table style=\"width:100%;\">\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t\n", '\t' <repeats 12 times>, "\n\t\t\t\t"..., ecode=0x133dc3aa "N",      mstart=0x13708600 "<html>\n<head>\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\" />\n<meta name=\"viewport\" content=\"width=device-width, initial-scale=1, maximum-scale=1\">\n\n</head>\n\n<body style=\"padding"..., markptr=0x0, offset_top=4, md=0x7fffb4b1b620, ims=0, eptrb=0x0, flags=0, rdepth=17680) at /home/farmer/srcs/php-5.3.4/ext/pcre/pcrelib/pcre_exec.c:470 470 { 

If I do a 'bt', I just get this "match" statement over and over again. Here are the first few:

#1  0x000000000046ef1e in match (     eptr=0x1370a887 "/td>\n\t\t\t\t\t\t\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t</table>\n\t\t\t</td>\n\t\t</tr>\n\t\t\n\t\t<tr>\n\t\t\t<td style=\"border-top:3px solid #efefef; padding:20px 10px;\">\n\t\t\t\t<table style=\"width:100%;\">\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t\n", '\t' <repeats 12 times>, "\n\t\t\t\t"..., ecode=0x133dc3a5 "_", mstart=<value optimized out>, markptr=<value optimized out>,      offset_top=<value optimized out>, md=0x7fffb4b1b620, ims=0, eptrb=0x0, flags=0, rdepth=17679) at /home/farmer/srcs/php-5.3.4/ext/pcre/pcrelib/pcre_exec.c:803 #2  0x0000000000479789 in match (     eptr=0x1370a887 "/td>\n\t\t\t\t\t\t\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t</table>\n\t\t\t</td>\n\t\t</tr>\n\t\t\n\t\t<tr>\n\t\t\t<td style=\"border-top:3px solid #efefef; padding:20px 10px;\">\n\t\t\t\t<table style=\"width:100%;\">\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t\n", '\t' <repeats 12 times>, "\n\t\t\t\t"..., ecode=0x133dc5ca "V\002%\033U\002.", mstart=<value optimized out>,      markptr=<value optimized out>, offset_top=<value optimized out>, md=0x7fffb4b1b620, ims=0, eptrb=0x0, flags=0, rdepth=<value optimized out>)     at /home/farmer/srcs/php-5.3.4/ext/pcre/pcrelib/pcre_exec.c:1623 #3  0x000000000046ef1e in match (     eptr=0x1370a886 "</td>\n\t\t\t\t\t\t\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t</table>\n\t\t\t</td>\n\t\t</tr>\n\t\t\n\t\t<tr>\n\t\t\t<td style=\"border-top:3px solid #efefef; padding:20px 10px;\">\n\t\t\t\t<table style=\"width:100%;\">\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t\n", '\t' <repeats 12 times>, "\n\t\t\t"..., ecode=0x133dc3a5 "_", mstart=<value optimized out>, markptr=<value optimized out>,      offset_top=<value optimized out>, md=0x7fffb4b1b620, ims=0, eptrb=0x0, flags=0, rdepth=17677) at /home/farmer/srcs/php-5.3.4/ext/pcre/pcrelib/pcre_exec.c:803 #4  0x0000000000479789 in match (     eptr=0x1370a886 "</td>\n\t\t\t\t\t\t\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t</table>\n\t\t\t</td>\n\t\t</tr>\n\t\t\n\t\t<tr>\n\t\t\t<td style=\"border-top:3px solid #efefef; padding:20px 10px;\">\n\t\t\t\t<table style=\"width:100%;\">\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t\n", '\t' <repeats 12 times>, "\n\t\t\t"..., ecode=0x133dc5ca "V\002%\033U\002.", mstart=<value optimized out>,      markptr=<value optimized out>, offset_top=<value optimized out>, md=0x7fffb4b1b620, ims=0, eptrb=0x0, flags=0, rdepth=<value optimized out>)     at /home/farmer/srcs/php-5.3.4/ext/pcre/pcrelib/pcre_exec.c:1623 #5  0x000000000046ef1e in match (     eptr=0x1370a885 "\t</td>\n\t\t\t\t\t\t\n\t\t\t\t\t</tr>\n\t\t\t\t\t\n\t\t\t\t</table>\n\t\t\t</td>\n\t\t</tr>\n\t\t\n\t\t<tr>\n\t\t\t<td style=\"border-top:3px solid #efefef; padding:20px 10px;\">\n\t\t\t\t<table style=\"width:100%;\">\n\t\t\t\t\t<tr>\n\t\t\t\t\t\t\n", '\t' <repeats 12 times>, "\n\t\t"..., ecode=0x133dc3a5 "_", mstart=<value optimized out>, markptr=<value optimized out>,      offset_top=<value optimized out>, md=0x7fffb4b1b620, ims=0, eptrb=0x0, flags=0, rdepth=17675) at /home/farmer/srcs/php-5.3.4/ext/pcre/pcrelib/pcre_exec.c:803 

I'm really at a loss here. I've tried running the command in strace, and it seems to be seg-faulting in the PEAR Mail_Queue library. Here's the end of the strace (some identifying information has been removed):

lstat("/src/prod/releases/20121025202143/www/application/pear/MDB2/Driver/mysql.php", {st_mode=S_IFREG|0770, st_size=55723, ...}) = 0 lstat("/src/prod/releases/20121025202143/www/application/pear/MDB2/Driver", {st_mode=S_IFDIR|0770, st_size=4096, ...}) = 0 lstat("/src/prod/releases/20121025202143/www/application/pear/MDB2", {st_mode=S_IFDIR|0770, st_size=4096, ...}) = 0 lstat("/src/prod/releases/20121025202143/www/application/pear", {st_mode=S_IFDIR|0770, st_size=4096, ...}) = 0 open("/src/prod/releases/20121025202143/www/application/pear/MDB2/Driver/mysql.php", O_RDONLY) = 6 fstat(6, {st_mode=S_IFREG|0770, st_size=55723, ...}) = 0 fstat(6, {st_mode=S_IFREG|0770, st_size=55723, ...}) = 0 fstat(6, {st_mode=S_IFREG|0770, st_size=55723, ...}) = 0 mmap(NULL, 55723, PROT_READ, MAP_SHARED, 6, 0) = 0x2b49f7bb4000 brk(0x987b000)                          = 0x987b000 brk(0x98bb000)                          = 0x98bb000 munmap(0x2b49f7bb4000, 55723)           = 0 close(6)                                = 0 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 6 fcntl(6, F_SETFL, O_RDONLY)             = 0 fcntl(6, F_GETFL)                       = 0x2 (flags O_RDWR) fcntl(6, F_GETFL)                       = 0x2 (flags O_RDWR) fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK)    = 0 connect(6, {sa_family=AF_INET, sin_port=htons(3306), sin_addr=inet_addr("IP ADDRESS TO DB SERVER REMOVED")}, 16) = -1 EINPROGRESS (Operation now in progress) fcntl(6, F_SETFL, O_RDWR)               = 0 poll([{fd=6, events=POLLIN|POLLPRI}], 1, 60000) = 1 ([{fd=6, revents=POLLIN}]) setsockopt(6, SOL_SOCKET, SO_RCVTIMEO, "\2003\341\1\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 0 setsockopt(6, SOL_SOCKET, SO_SNDTIMEO, "\2003\341\1\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 0 setsockopt(6, SOL_IP, IP_TOS, [8], 4)   = 0 setsockopt(6, SOL_TCP, TCP_NODELAY, [1], 4) = 0 setsockopt(6, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0 poll([{fd=6, events=POLLIN}], 1, 60000) = 1 ([{fd=6, revents=POLLIN}]) read(6, "C\0\0\0\n5.1.37-1ubuntu5.5-log\0Q\257\3\0009"..., 16384) = 71 write(6, "=\0\0\1\205\242\2\0\0\0\0@\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 65) = 65 read(6, "\7\0\0\2\0\0\0\2\0\0\0", 16384) = 11 poll([{fd=6, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout) write(6, "\10\0\0\0\2xxxxx", 12)      = 12 read(6, "\7\0\0\1\0\0\0\2\0\0\0", 16384) = 11 lstat("/src/prod/current/www/application/pear/Mail/mime.php", {st_mode=S_IFREG|0770, st_size=50252, ...}) = 0 lstat("/src/prod/current/www/application/pear/Mail", {st_mode=S_IFDIR|0770, st_size=4096, ...}) = 0 lstat("/src/prod/current/www/application/pear", {st_mode=S_IFDIR|0770, st_size=4096, ...}) = 0 lstat("/src/prod/current/www/application", {st_mode=S_IFDIR|0776, st_size=4096, ...}) = 0 --- SIGSEGV (Segmentation fault) @ 0 (0) --- +++ killed by SIGSEGV +++ Segmentation fault 

If anyone has any ideas or pointers to lines of investigation, I'd truly appreciate it. I'm really stumped on this one.

Answer 1

I usually use xdebug if I encounter a segfault. I prefer to use the following line:

php -d xdebug.auto_trace=ON -d xdebug.trace_output_dir=mytracedir/ myscript.php 

With the trace-output under mytracedir/, you can easily figure out the approximate php line, which breaks the script. Then rewrite the line in one statement per line and use error_log() for debugging vars. If you use __destruct() or __toString() it could get a bit more tricky there.

See here for more informations.

(If the extension is not loaded, but installed, try to add php -dzend_extension=xdebug.so [...])

Answer 2

The debugger already tells you what happens:

... rdepth=17680) at /home/farmer/srcs/php-5.3.4/ext/pcre/pcrelib/pcre_exec.c:470 

See pcre.recursion_limit setting:

pcre.recursion_limit - PCRE's recursion limit. Please note that if you set this value to a high number you may consume all the available process stack and eventually crash PHP (due to reaching the stack size limit imposed by the Operating System).

Reduce its value so that it does not crash any longer for the quick fix. You will then see your HTML not fully processed by the regular expression.

Improve the expression to not rely on recursion that much. You probably just want to use a HTML parser.