Too many open files when using Memcached for sessions

Question

I'm using ubuntu + nginx + php5-fpm + symfony2, and I recently switched from files to Memcached (php5-memcached) for sessions. Since then, I've started to get 500 response code errors after a while. This is what it looks like in the error logs (failed to open stream: Too many open):

"PHP message: PHP Warning:  simplexml_load_file(.../vendor/friendsofsymfony/user-bundle/FOS/UserBundle/Resources/config/doctrine/User.orm.xml): failed to open stream: Too many open files in .../vendor/doctrine/orm/lib/Doctrine/ORM/Mapping/Driver/XmlDriver.php on line 736
PHP message: PHP Warning:  simplexml_load_file(): I/O warning : failed to load external entity ".../vendor/friendsofsymfony/user-bundle/FOS/UserBundle/Resources/config/doctrine/User.orm.xml" in .../vendor/doctrine/orm/lib/Doctrine/ORM/Mapping/Driver/XmlDriver.php on line 736
PHP message: PHP Warning:  include(.../vendor/doctrine/common/lib/Doctrine/Common/Persistence/Mapping/MappingException.php): failed to open stream: Too many open files in .../vendor/composer/ClassLoader.php on line 382

After some research, I've found out that it seems to be related to the usage of Memcached for sessions:

1. Those are the php5-fpm processes running:

   ps aux | grep php

   root 3233 0.0 0.2 207464 16164 ? Ss Mar16 0:02 php-fpm: master process (/etc/php5/fpm/php-fpm.conf)
   www-data 3236 0.0 0.8 238796 65796 ? S Mar16 0:50 php-fpm: pool www
   www-data 3646 0.0 0.7 231176 57808 ? S Mar16 0:42 php-fpm: pool www
   www-data 7503 0.1 0.7 234820 59920 ? S 11:52 0:34 php-fpm: pool www
   ubuntu 8224 0.0 0.0 10436 860 pts/0 S+ 16:53 0:00 grep --color=auto php

2.Checking any of the php5-fpm childs with lsof, it shows, among other things that there were 1k+ open file descriptors pointing to the Memcache instance with an established connection:

sudo lsof -p 3236

…
php5-fpm 3236 www-data  672u  IPv4             107781      0t0    TCP hostname:54403->memcacheIp:11211 (ESTABLISHED)
php5-fpm 3236 www-data  673u  IPv4             108827      0t0    TCP hostname:54411->memcacheIp:11211 (ESTABLISHED)
php5-fpm 3236 www-data  674u  IPv4             107800      0t0    TCP hostname:54418->memcacheIp:11211 (ESTABLISHED)
... 

Over one thousand entries like that, which I think starts failing when it arrives to 1024, which is the limit of open file descriptors:

ulimit  -n
1024

The thing is that this is a QA environment which is not used for more than 1 or 2 concurrent users, so there is no way all those open file descriptors are active sessions. I noticed that every time I make a request to the symfony2 application, a new socket connection to the memcache server is opened, but then it's never closed. So, eventually it reaches the limit at starts failing. I thought might be something related with timeout connections or something like that but haven't found anything so far.

Restarting php5-fpm seems to "solve" the problem, but only until another 1k requests are made.

I've temporarily switched back to files and the problem disappears. Also I've tried using Memcache (php5-memcache) instead of Memcached and the problem is solved but I'd rather use php5-memcached as it seems better maintained.

Any ideas of how to solve this problem?

Thank you very much!

If it helps, the specific php version that I'm using:

PHP 5.5.9-1ubuntu4.6 (cli) (built: Feb 13 2015 19:17:11) 
libmemcached version => 1.0.8

Answer 1

Ok, I've found the solution thanks to this:

http://php.net/manual/en/memcached.construct.php (see @Tobias comment)

and this https://gist.github.com/K-Phoen/4327229 (see @cmenning comment)

I was using Memcached with a sessions.yml as follows:

session.memcached:
    class: Memcached
    arguments:
       persistent_id: %session_memcached_prefix%
    calls:
        - [ addServer, [ %session_memcached_host%, %session_memcached_port% ]]

session.handler.memcached:
    class:     Symfony\Component\HttpFoundation\Session\Storage\Handler\MemcachedSessionHandler
    arguments: [@session.memcached, { prefix: %session_memcached_prefix%, expiretime: %session_memcached_expire% }]

It turns out that if you provide a persistent_id argument for Memcached it would persist the connection between requests. From php.net:

persistent_id

By default the Memcached instances are destroyed at the end of the request. To create an instance that persists between requests, use persistent_id to specify a unique ID for the instance. All instances created with the same persistent_id will share the same connection.

The problem is that if you use a persistent_id (Which is better for performance), you have to check if there are already servers added before calling addServer, otherwise it would add a new server & create a new connection, even though it's the same server (it doesn't check for dupes. Something like this:

        $instance = new Memcached($persistent_id);

        // Add server if no connections listed. 
        if (!count($instance->getServerList())) {
            $instance->addServers($server);
        }

But for that you'd need to create a wrapper class of Memcached. The easy fix is just to comment out this lines, and avoid using persistent connections for the moment:

    #arguments:
    #  persistent_id: %session_memcached_prefix%

This solves the issue, even though it's not ideal. I hope it helps you to save some time and avoid a headache! Thanks everyone for your help.