How to add an item to a memcached list atomically (in Python)

Question

Behold my simple Python memcached code below:

import memcache
memcache_client = memcache.Client(['127.0.0.1:11211'], debug=True)
key = "myList"
obj = ["A", "B", "C"]
memcache_client.set(key, obj)

Now, suppose I want to append an element "D" to the list cached as myList, how can I do it atomically?

I know this is wrong because it is not atomic:

memcache_client.set(key, memcache_client.get(key) + ["D"])

The above statement contains a race condition. If another thread executes this same instruction at the exact right moment, one of the updates will get clobbered.

How can I solve this race condition? How can I update a list or dictionary stored in memcached atomically?

Answer 1

Here's the corresponding function of the python client API

https://cloud.google.com/appengine/docs/python/memcache/clientclass#Client_cas

Also here's a nice tutorial by Guido van Rossum. Hope he'd better explain python stuff than I ;)

Here's how the code should look like in your case:

memcache_client = memcache.Client(['127.0.0.1:11211'], debug=True)
key = "myList"
while True: # Retry loop, probably it should be limited to some reasonable retries
  obj = memcache_client.gets(key)
  assert obj is not None, 'Uninitialized object'
  if memcache_client.cas(key, obj + ["D"]):
    break

The whole workflow remains the same: first you fetch a value (w/ some internal information bound to a key), then modify the fetched value, then attempt to update it in the memcache. The only difference that the value (actually, key/value pair) is checked that it hasn't been changed simultaneosly from a parallel process. In the latter case the call fails and you should retry the workflow from the beginning. Also, if you have a multi-threaded application, then each memcache_client instance likely should be thread-local.

Also don't forget that there're incr() and decr() methods for simple integer counters which are "atomic" by their nature.