600+ memcache req/s problems - help!

Question

I am running memcached on my server and when it hits 600+ req/s it becomes unstable and causes a big load of problems. It appears when the request rate gets that high, my PHP applications at random times are unable to connect to the memcache server, causing slow load times which makes nginx and php-fpm freak out and I receive a bunch of 104: Connection reset by peer errors in my nginx logs.

I would like to point out that in my memcache server I have 'hot objects' - objects that at times receive 90% of the memcache requests. I also noticed when so many requests hit a single object, it slightly adds a little more load time to the overall page (when it manages to load).

I would greatly appreciate any help to this problem. Thanks so much!

Answer 1

Switch away from using TCP sockets and going to UNIX sockets (assuming you are on a unix based server)

Start memcached with a socket enabled: Add -s /tmp/memcached.socket to your memcached startup line (Note, sockets disables networking support)

Then in PHP, connect using persistent connections, and to the new memcache socket:

$memcache_obj = new Memcache;
$memcache_obj->pconnect('unix:///tmp/memcached.socket', 0);

Another recommendation, if you have multiple "types" of cached objects, start a memcached instance for each "type" and distribute your hot items amongst them.

Drupal does this, you can see how their config file and memcached init is setup here.

Also, it sounds to me like your memcached timeout is set WAY to high. If it's anything above 1 or 2 seconds, you can lock scripts up. The timeout should be reached, and the script should default to retrieving the object via another method (SQL, file, etc)

The other thing is verify that your memcache isn't being put into a swap file, if your cache is smaller than your average free ram, try starting memcache with the -k option, this will force it's cache to always stay in ram and can't be swapped.

If you have a multi-core server, also make sure memcached is compiled with thread support, and enable it using -t <numcores>

Answer 2

600 requests per second is profoundly low for memcached.

If you're establishing a connection for every request, you'll spend more time connecting than requesting and burn through your ephemeral ports very rapidly which might be the problem you're seeing.