PHP Opcode cache locking Apache . Maybe Symfony2 - Doctrine2 related

Question

I developped about 10 different websites, hosted on a single server with spec mentionned below.

Story:

Everything was running fine, until I decided to integrate a PHP Opcode cache into the system. I first tried with APC, but the same problem appeared with Xcache, so i don't think it is related to the cache program itself.

The system remain stable for a period, varying from a day to a week, and crashes at various hours, but mainly at night around 23h-05h. If I restart the httpd service, the system is stable again, during the same period (1 day to 1 week), and crashes again etc...

Bug Report:

Here is a report of my httpd global log during the last crash:

[Thu Feb 18 20:00:11.270997 2016] [core:notice] [pid 24956:tid 139940499228480] AH00052: child pid 4522 exit signal Aborted (6)
httpd: hostip.c:693: Curl_resolv_unlock: Assertion `dns && (dns->inuse>0)' failed.
[Thu Feb 18 20:08:38.793218 2016] [core:notice] [pid 24956:tid 139940499228480] AH00052: child pid 6246 exit signal Aborted (6)
httpd: hostip.c:693: Curl_resolv_unlock: Assertion `dns && (dns->inuse>0)' failed.
[Thu Feb 18 22:12:33.576308 2016] [core:notice] [pid 24956:tid 139940499228480] AH00052: child pid 8362 exit signal Aborted (6)
httpd: hostip.c:693: Curl_resolv_unlock: Assertion `dns && (dns->inuse>0)' failed.
[Thu Feb 18 22:40:07.297428 2016] [core:notice] [pid 24956:tid 139940499228480] AH00052: child pid 10224 exit signal Aborted (6)
[Thu Feb 18 23:00:40.526867 2016] [core:warn] [pid 24956:tid 139940499228480] AH00045: child process 10846 still did not exit, sending a SIGTERM
...

Note that the server became blocked around 22:41:xx . I restarted the server at 23:00:xx, which is related to the last line posted here.

It could be be related to curl, I'm using it a lot, specialy PHP curl_multi, in order to speed my API calls by launching them simultaneously. But the same error happens also without any PHP opcode cache installed (or disabled), but it does not crash the server down.

Server State:

When a "crash" occurs, the system can still serve any txt, image etc... file, but it can not serve any PHP file. The server is locked, and when taking a look at the "Apache Server Status", there is maybe a hundred request in a "W" state, increasing with each incoming request.

Cache Implementation:

As stated above, I tried using APC and xCache, but both of them produced the same problem. My PHP is a compiled version (that I did not compiled myself). There is a Varnish cache above all the websites, only caching a few time-hungry pages. I'm using Symfony2 with Doctrine2, with a link between Doctrine2 and xcache/apc via a Symfony2 config:

doctrine:
    orm:
        auto_generate_proxy_classes: prod
        auto_mapping: true
        metadata_cache_driver: xcache
        result_cache_driver: xcache
        query_cache_driver: xcache

It seems to allow Doctrine2 to cache entity object and produce way better performances (only enabling cache was not enough for Doctrine2)

Specifications:

- Debian 7.8
- Apache 2.4.12
- Mysql 
- PHP 5.4.38 (compiled version)
- Varnish
- Symfony 2.4.x

Any hint or help will be greatly welcomed, as I've been looking for a solution for months, and running a PHP-Symfony2 website is alot slower without any opcode cache

Answer 1

Do you have any specific cron jobs that are running around the time the server crashes?

In any case, if you are seeing Curl_resolv_unlock: Assertion 'dns && (dns->inuse>0)' failed in the logs, this means you have a "debug build" of libcurl active in PHP which is not good.

Your comment mentioned PHP says Apache 2.0 Handler, so it is running as an Apache module.

When that debug assertion hits in libcurl, it causes the process (PHP) to terminate. Basically what I think is happening is that your PHP process for Apache (which handles all PHP requests) is dying. This is why you can still serve static resources but PHP processes keep stacking up.

cURL 7.26.0 is fairly old (May 2012). I'd suggest installing a newer version of libcurl, and then recompiling PHP with it and make sure cURL is not a debug build and see if this helps.

Answer 2

My server API is "Apache 2.0 Handler"

You should definitely check out if a simillar behavior occurs under PHP-FPM. Apache SAPI means it's not separated from httpd's memory space as using FPM pool process.

I suspect you encounter on some kind of memory leak that cannot be easily traced. Changing to FPM will let you also to automatically restart responder after some time (requests amount) which is far more elegant and flexible than croning and checking if process works well.

Honestly, it's been some time since I had seen PHP connected by Apache SAPI. :)