Update DynamoDB Atomic Counter with Python / Boto

Question

I am trying to update an atomic count counter with Python Boto 2.3.0, but can find no documentation for the operation.

It seems there is no direct interface, so I tried to go to "raw" updates using the layer1 interface, but I was unable to complete even a simple update.

I tried the following variations but all with no luck

dynoConn.update_item(INFLUENCER_DATA_TABLE, 
                     {'HashKeyElement': "9f08b4f5-d25a-4950-a948-0381c34aed1c"}, 
                     {'new': {'Value': {'N':"1"}, 'Action': "ADD"}})    

dynoConn.update_item('influencer_data', 
                     {'HashKeyElement': "9f08b4f5-d25a-4950-a948-0381c34aed1c"}, 
                     {'new': {'S' :'hello'}})                                 

dynoConn.update_item("influencer_data", 
                     {"HashKeyElement": "9f08b4f5-d25a-4950-a948-0381c34aed1c"},
                     {"AttributesToPut" : {"new": {"S" :"hello"}}})      

They all produce the same error:

  File "/usr/local/lib/python2.6/dist-packages/boto-2.3.0-py2.6.egg/boto/dynamodb/layer1.py", line 164, in _retry_handler
    data)
boto.exception.DynamoDBResponseError: DynamoDBResponseError: 400 Bad Request
{u'Message': u'Expected null', u'__type': u'com.amazon.coral.service#SerializationException'}

I also investigated the API docs here but they were pretty spartan.

I have done a lot of searching and fiddling, and the only thing I have left is to use the PHP API and dive into the code to find where it "formats" the JSON body, but that is a bit of a pain. Please save me from that pain!

Answer 1

Sorry, I misunderstood what you were looking for. You can accomplish this via layer2 although there is a small bug that needs to be addressed. Here's some Layer2 code:

>>> import boto
>>> c = boto.connect_dynamodb()
>>> t = c.get_table('counter')
>>> item = t.get_item('counter')
>>> item
{u'id': 'counter', u'n': 1}
>>> item.add_attribute('n', 20)
>>> item.save()
{u'ConsumedCapacityUnits': 1.0}
>>> item  # Here's the bug, local Item is not updated
{u'id': 'counter', u'n': 1}
>>> item = t.get_item('counter')  # Refetch item just to verify change occurred
>>> item
{u'id': 'counter', u'n': 21}

This results in the same over-the-wire request as you are performing in your Layer1 code, as shown by the following debug output.

2012-04-27 04:17:59,170 foo [DEBUG]:StringToSign:
POST
/

host:dynamodb.us-east-1.amazonaws.com
x-amz-date:Fri, 27 Apr 2012 11:17:59 GMT
x-amz-security-    token:<removed> ==
x-amz-target:DynamoDB_20111205.UpdateItem

{"AttributeUpdates": {"n": {"Action": "ADD", "Value": {"N": "20"}}}, "TableName": "counter", "Key": {"HashKeyElement": {"S": "counter"}}}

If you want to avoid the initial GetItem call, you could do this instead:

>>> import boto
>>> c = boto.connect_dynamodb()
>>> t = c.get_table('counter')
>>> item = t.new_item('counter')
>>> item.add_attribute('n', 20)
>>> item.save()
{u'ConsumedCapacityUnits': 1.0}

Which will update the item if it already exists or create it if it doesn't yet exist.

Answer 2

For those looking for the answer I have found it. First IMPORTANT NOTE, I am currently unaware of what is going on BUT for the moment, to get a layer1 instance I have had to do the following:

import boto
AWS_ACCESS_KEY=XXXXX
AWS_SECRET_KEY=YYYYY
dynoConn = boto.connect_dynamodb(AWS_ACCESS_KEY, AWS_SECRET_KEY)
dynoConnLayer1 = boto.dynamodb.layer1.Layer1(AWS_ACCESS_KEY, AWS_SECRET_KEY) 

Essentially instantiating a layer2 FIRST and THEN a layer 1. Maybe Im doing something stupid but at this point Im just happy to have it working.... I'll sort the details later. THEN...to actually do the atomic update call:

dynoConnLayer1.update_item("influencer_data", 
                    {"HashKeyElement":{"S":"9f08b4f5-d25a-4950-a948-0381c34aed1c"}},
                    {"direct_influence":
                        {"Action":"ADD","Value":{"N":"20"}}
                    }
                );

Note in the example above Dynamo will ADD 20 to what ever the current value is and this operation will be atomic meaning other operations happening at the "same time" will be correctly "scheduled" to happen after the new value has been established as +20 OR before this operation is executed. Either way the desired effect will be accomplished.

Be certain to do this on the instance of the layer1 connection as the layer2 will throw errors given it expects a different set of parameter types.

Thats all there is to it!!!! Just so folks know, I figured this out using the PHP SDK. Takes a very short time to install and set up AND THEN when you do a call, the debug data will actually show you the format of the HTTP request body so you will be able to copy/model your layer1 parameters after the example. Here is the code I used to do the atomic update in PHP:

<?php 
    // Instantiate the class
    $dynamodb = new AmazonDynamoDB();

    $update_response = $dynamodb->update_item(array(
        'TableName' => 'influencer_data',
            'Key' => array(
                'HashKeyElement' => array(
                    AmazonDynamoDB::TYPE_STRING=> '9f08b4f5-d25a-4950-a948-0381c34aed1c'
                )
            ),
            'AttributeUpdates' => array(
                'direct_influence' => array(
                    'Action' => AmazonDynamoDB::ACTION_ADD,
                    'Value' => array(
                        AmazonDynamoDB::TYPE_NUMBER => '20'
                    )
                )
            )
    ));

    // status code 200 indicates success
    print_r($update_response);

?>

Hopefully this will help other up until the Boto layer2 interface catches up...or someone simply figures out how to do it in level2 :-)

Answer 3

I'm not sure this is truly an atomic counter, since when you increment the value of 1, another call call could increment the number by 1, so that when you "get" the value, it is not the value that you would expect.

For instance, putting the code by garnaat, which is marked as the accepted answer, I see that when you put it in a thread, it does not work:

class ThreadClass(threading.Thread):
    def run(self):
        conn = boto.dynamodb.connect_to_region(aws_access_key_id=os.environ['AWS_ACCESS_KEY'], aws_secret_access_key=os.environ['AWS_SECRET_KEY'], region_name='us-east-1')
        t = conn.get_table('zoo_keeper_ids')
        item = t.new_item('counter')
        item.add_attribute('n', 1)
        r = item.save() #- Item has been atomically updated!
        # Uh-Oh! The value may have changed by the time "get_item" is called!
        item = t.get_item('counter') 
        self.counter = item['n']
        logging.critical('Thread has counter: ' + str(self.counter))

tcount = 3
threads = []
for i in range(tcount):
    threads.append(ThreadClass())

# Start running the threads:
for t in threads:
    t.start()

# Wait for all threads to complete:
for t in threads:
    t.join()

#- Now verify all threads have unique numbers:
results = set()
for t in threads:
    results.add(t.counter)

print len(results)
print tcount
if len(results) != tcount:
    print '***Error: All threads do not have unique values!'
else:
    print 'Success!  All threads have unique values!'

Note: If you want this to truly work, change the code to this:

def run(self):
    conn = boto.dynamodb.connect_to_region(aws_access_key_id=os.environ['AWS_ACCESS_KEY'], aws_secret_access_key=os.environ['AWS_SECRET_KEY'], region_name='us-east-1')
    t = conn.get_table('zoo_keeper_ids')
    item = t.new_item('counter')
    item.add_attribute('n', 1)
    r = item.save(return_values='ALL_NEW') #- Item has been atomically updated, and you have the correct value without having to do a "get"!
    self.counter = str(r['Attributes']['n'])
    logging.critical('Thread has counter: ' + str(self.counter))

Hope this helps!