Tracking down a segmentation fault 11 to large PHP code base

Question

Ok so have a fairly huge in house system. We have a nice friendly host who generally work with us keeping things working if something ever goes wrong, or at least points us in the right direction.

A while ago we had to take a copy of the code and host it on a clients external dedicated server, hosted with Rackspace. Considering everything was working fine for a number of months, all of a sudden the server kept throwing hissy-fits and segmentation faults eg

[notice] child pid 8628 exit signal Segmentation fault (11)

Even though no code had been changed, the hosts have said its probably a PHP issue and the devs (us) would have to look into it. I haven't even seen a problem, just that at certain times when the apache logs rotate (late at night) this is causing it to fall over.

I got them to enable core dumps, and I am armed with a bunch of them, and Fixing Apache Segmentation Faults Caused by PHP, but rackspace seem to have moved all the php and apache source dirs to god knows where, I cant find .gdbinit anywhere on the server using 'find', and about all I can get out of gdb is

Program terminated with signal 11, Segmentation fault.
#0  0x00002b8d6fc36c60 in ?? ()

...which couldnt be less helpful if it tried!!!

Any ideas on where to start with tracking down this supposed problem in my code?

Answer 1

You can start by adding PID to apache logs and see which one dies after what... You will see if its the same request atleast.

And maybe you can see what was updated just before crashing started and maybe you can rollback to previous configuration if some modules where updated and can't live without them.

Answer 2

If you know how to debug application cores using GDB, here are the source RPMs and RPMs.

I'd also suggest you run "strings" on the application core. This will not give you exact problem but in many cases, it gives something useful.

HTH