How to convert from DynamoDB wire protocol to native Python object manually with boto3?

Question

I have a Lambda that is being triggered by a DynamoDB stream. The Lambda does some processing and then creates a notification on a topic in SNS. Ideally I would like to include the entire new document in the notification that goes out to SNS so that downstream clients don't have to hit DynamoDB to get the data.

The problem I'm running into is that the data coming from the DynamoDB stream is in DynamoDB wire format (the maps include the data type as a key). When I send out the notification to downstream clients I don't want them to have to understand DynamoDB wire format to parse the message (for example if I switch to a new underlying data store I would then have to recreate that format).

Obviously the boto3 client is capable of parsing this format into a Python object, is there a way for me to access the parser on my own? As far as I can tell it gets called as part of fetching data from DynamoDB but I can't find a way to call it on my own.

Answer 1

I have a similar situation and I used the following an approach like this:

from boto3.dynamodb.types import TypeDeserializer

deser = TypeDeserializer()

...
<in handler>
    for record in event['Records']:
        old = record['dynamodb'].get('OldImage')
        new = record['dynamodb'].get('NewImage')
        if old:
            d = {}
            for key in old:
                d[key] = deser.deserialize(old[key])

This approach works for me. The resulting dictionary d contains the converted object rather than the wire-format version passed to the handler.

Answer 2

Using TypeDeserializer didn't work for me for some reason. This utility did, however:

https://github.com/Alonreznik/dynamodb-json

Those not wanting to deal with emulating pip install inside a Lambda function, you can just copy and paste this source file (containing the loads() function) into your own code:

https://github.com/Alonreznik/dynamodb-json/blob/master/dynamodb_json/json_util.py