Memcached: auto-discovery python support on AWS Elasticache?

Question

I started to use AWS Elasticache with my django web app.

I started by setting the cache location to the unique endpoint using the auto-discovery feature, but it doesn't seems to work.

I'm using pylibmc (1.2.2) and django-pylibmc-sasl (0.2.4) to connect to memcached from python.

Does the auto-discovery feature work on these clients? How can I enable it?

Answer 1

Quick answer

Yes for django: django-elasticache

Long Answer

ElastiCache provides memcached interface so there are three solution of using it:

1. Memcached configured with location = Configuration Endpoint.

In this case your application will randomly connect to nodes in cluster and cache will be used with not optimal way. At some moment you will be connected to first node and set item. Minute later you will be connected to another node and will not able to get this item.

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache',
        'LOCATION': 'cache.gasdbp.cfg.use1.cache.amazonaws.com:11211',
    }
}

2. Memcached configured with all nodes.

It will work fine, memcache client will separate items between all nodes and will balance loading on client side. You will have problems only after adding new nodes or delete old nodes. In this case you should add new nodes manually and don't forget update your app after all changes on AWS.

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache',
        'LOCATION': [
            'cache.gqasdbp.0001.use1.cache.amazonaws.com:11211',
            'cache.gqasdbp.0002.use1.cache.amazonaws.com:11211',
        ]
    }
}

3. Use django-elasticache.

It will connect to cluster and retrieve ip addresses of all nodes and configure memcached to use all nodes.

CACHES = {
    'default': {
        'BACKEND': 'django_elasticache.memcached.ElastiCache',
        'LOCATION': 'cache-c.draaaf.cfg.use1.cache.amazonaws.com:11211',
    }
}

Difference between setup with nodes list (django-elasticache) and connection to only one configuration Endpoint (using dns routing) you can see on this graph:

Answer 2

I used the PyLibMC binding which doesn't seems to support auto-discovery.

The Memcached backend built-in with Django and used in the documentation is working well with the unique endpoint provided by Elasticache.

Now everything's running fine and I improved a lot my response time with Memcached.