memcached slows down website

Question

I have a website that is being served by nginx and django.

My staging.py contains CACHE and middleware settings correctly. You can take a look at nginx.conf and the nginx conf file related to the site. I have confirmed that memcached is running through ngrep -d any port 11211.

I turned on caching for the whole site, and wanted to see the performance by doing ab -n 1000 -c 10 http://site.com

With caching turned off, I get:

Concurrency Level:      10
Time taken for tests:   10.276 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      11695000 bytes
HTML transferred:       11559000 bytes
Requests per second:    97.32 [#/sec] (mean)
Time per request:       102.759 [ms] (mean)
Time per request:       10.276 [ms] (mean, across all concurrent requests)
Transfer rate:          1111.43 [Kbytes/sec] received

With caching turned on, I get:

Concurrency Level:      10
Time taken for tests:   12.277 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      11695000 bytes
HTML transferred:       11559000 bytes
Requests per second:    81.45 [#/sec] (mean)
Time per request:       122.771 [ms] (mean)
Time per request:       12.277 [ms] (mean, across all concurrent requests)
Transfer rate:          930.26 [Kbytes/sec] received

My website is a blog that is pulling posts from a database - nothing exotic.

I'd be grateful if someone could let me know why the site is actually slowing down with memcached. You can see that the "Requests per second" actually drops when I use memcached!

However, running memcached-top gave me no hits when I ran ab (though the read and write counters went up during the test). I have memory available and memcached is not hogging up memory.

EDIT
I ran memcached -vv and got some results. You can see that the the memcached prints out a "STORED" the first time, and then does not seems to send it from the cache (not sure about this). Now I am even more confused. Perhaps the memcached & the django interface is working, but the end result is that it better off to not run memcached?

Answer 1

Trewq, whole lot of different things could be going wrong. You said your machine isn't paging but get requests don't come back even though memcache STORED the result.

My theories: too short of timeouts, bad driver, and possibly wrong CPU arch (x86 vs _64)

Timeouts

Usually in the -vv output ( might be -vvv ) the SET line will have syntax like command, key, value, and a timeout. Very small timeout's might be the problem with memcache storing and then almost immediately flushing the value out.

<command name> <key> <flags> <exptime> <bytes> [noreply]\r\n - https://github.com/memcached/memcached/blob/master/doc/protocol.txt

Driver

Also, there might be an issue with the memcache driver/api you're using as mc should never block that long. You can check your memcache service state by doing something like this http://code.google.com/p/memcached/wiki/NewConfiguringServer#Inspecting_Running_Configuration before and after doing a bechmark run.

Key auditing

A while back I wrote the script in this question Setting smaller buffer size for sys.stdin? to audit the output of memcache -vv to see how balanced GETs were to SETs. It's been a while but I believe it might useful to you with some fixes.

It's not mentioned in the wiki for stat but there are stat values to help you figure out if your cache is balanced - https://github.com/memcached/memcached/blob/master/doc/protocol.txt#L409

Super ideal is 9/10 requests are hits to 1 miss, reality is more likely 6/10 hits to requests, and anything below 60% is wasting memory.