What's the simplest way to get a dump of all memcached keys into a file?

Question

This is from just a single memcached server with around 20M keys (no expiry) and around 2G of data.

What's the easiest way to get a dump of all the key/value pairs into a flat file? I first looked at the java net.spy.memcached.MemcachedClient, but this client does not support getting all keys (I think). If I had a list of all keys (which I don't), I could easily use this client to get all of the values.

I know I can get all keys using some telnet commands (e.g., telnet localhost 11211; stats items; stats cachedump ), but it's not clear to me how to automate this robustly.

EDIT: Here's what I did to get this working on a toy memcached server on my machine. It seems to work but I only put two keys in memcached, so hopefully this method will scale ok:

shell commands:

sudo yum install memcached
sudo /etc/init.d/memcached restart # maybe unnecessary
sudo yum install php
sudo yum install php-pecl-memcache
sudo service httpd reload

php script, based on this:

<?php
$memcache = new Memcache();
$memcache->connect('127.0.0.1', 11211) or die ("Could not connect");
$list = array();
$allSlabs = $memcache->getExtendedStats('slabs');
$items = $memcache->getExtendedStats('items');
foreach($allSlabs as $server => $slabs) {
    foreach($slabs AS $slabId => $slabMeta) {
        if (!is_int($slabId)) {
            continue;
        }
        $cdump = $memcache->getExtendedStats('cachedump', (int) $slabId, 100000000);
        foreach($cdump AS $server => $entries) {
            if ($entries) {
                foreach($entries AS $eName => $eData) {
                    print_r($eName);
                    print_r(":");
                    $val = $memcache->get($eName);
                    print_r($val);
                    print_r("\n");
                }
            }
        }
    }
}
?>

EDIT2: The above script does not seem to return all of the mappings. If I insert the line count($entries), it only returns a little over 50k, even with the limit parameter set to 100M, but executing stats items from telnet shows over 5M entries. Does anyone know why this might be the case?

EDIT3: This link suggests that cachedump doesn't get all keys from memcached. I've hit a limit of around 50k keys that are returned either by cachedump, this PHP script, or a perl script similar to one in the link provided by Zach Bonham. Is there any way around this?

Answer 1

disclaimer: I don't know what I"m doing, just sounded like an interesting problem.

Did you see this article? "How to Dump Keys from Memcache" by Lars Windolf.

From the article:

Memcache itself provide the means to peak into the data. The protocol provides commands to peak into the data that is organized by slabs (categories of data of a given size range. There are some significant limitations though:

• You can only dump keys per slab class (keys with roughly the same content size)
• You can only dump one page per slab class (1MB of data)
• This is an unofficial feature that might be removed anytime.

Effectively, it requires some knowledge of how memcache stores data in memory (which I don't). You need to find each 'slab', then you can dump the keys for that slab, and then ultimately, the values for those keys.

There is a tools section in the article which uses various languages to dump at least the keys, but only the perl script dumps both keys and values.

Answer 2

memccat

Here is the script which I'm using to dump all the objects into corresponding files:

while read -r key; do
    [ -f "$key" ] || echo "get $key" | nc localhost 11211 > "$key.dump";
done < <(memcdump --server localhost)

It uses memcdump command which should be part of memcached utils.

For compressed objects, see: How to dump a compressed object for given key from Memcache?

memcdump

To dump a list of keys from a server, use memcdump/memdump tool, e.g.

memcdump --servers=localhost | tee my_keys.lst

To print the value of one item, use netcat:

echo "get 13456_-cache-some_object" | nc localhost 11211

To dump all objects into the screen via memcdump/memdump and netcat:

memcdump --servers=localhost | xargs -L1 -I% sh -c 'echo "get %" | nc localhost 11211'

memcached-tool

In the recent version of memcached there is also memcached-tool command, e.g.

memcached-tool localhost:11211 dump | less # dumps keys and values

Answer 3

There is a hardcoded limit of 2MB for the dump of a slab. Unless you rewrite the do_item_cachedump, you will not be able to get all keys out.