Memcached slowing down considerably under load on Symfony2+Doctrine2 application

Question

I've been doing load testing on my Symfony2 application and one of the bottlenecks I've identified appears to be Memcached.

I'm using memcached on the webserver to cache doctrine metadata and queries, and I'm using amazon's elasticache memcached as doctrine's result cache.

On a request without any load my end-to-end time is 300-350ms, and memcached accounts for about 10 of these ms

However, when under a load of 25 concurrent requests, my total response is slowed down to about 2500ms, and a large chunk of this appears to be memcached

34 of the 39 memcached queries are fetching classmetadata, and of these some are at .1ms and others are at 40 or 50ms, it seems to be pretty random.

I've tried increasing the max connections and cache size in the local memcache configuration file, is there anything else I can do to try and remove this bottleneck from my application?

Thanks a bunch-

Answer 1

Network I/O : Check if there is any issue with Network I/O. It can be due to multiple reasons:

• Network bandwidth is quite low.
• Memcached and your Application/Web server are in different zones and memcached is returning results perfectly, but the I/O time is huge.
• Size of the data returned from Memcached is quite large and lots of time is spent in serialization and de-seralization. If that is case, you can try to reduce the size by using a better mechanism for serialization and de-seralization.
  The inconsistency of the response time between 0.1ms to 40ms might be because of this reason.

Please verify the same.