Most efficient way to add prefix to Python dictionary keys

Question

So I find myself needing to add a prefix to a Python dictionary.

Basically what I want is for the user of this dictionary to be able to add a prefix on instantiation of the dictionary, in which case the dictionary holds the prefix and everytime a new key is added, it prepends the prefix. But I also want mutate the dictionary if for some reason the prefix is not provided or changed, this means that the old dictionary keys need to have the prefix prepended to them while keeping their respective values.

Use case:

Basically I'm finishing up the last apis of the MWS API. I built the api around the idea that every call needed to take specific parameters, like :

def get_report(self, marketplaceids):
    # Here I process marketplaceids which is a python list
    # and send the following to Amazon:

    MarketplaceIdList.Id.1: 123,
    MarketplaceIdList.Id.2: 345,
    MarketplaceIdList.Id.3: 4343

    # By doing this I eliminate the complexity of the arguments Amazon expects

Unfortunately the last two apis are harder to implement this way because they make use of a new "feature" Amazon introduced called Datatypes.

These "Datatypes" are nested structures. For example:

I want to call the CreateInboundShipment action from the InboundShipmentAPI,

The action takes the following arguments:

ShipmentId - String
InboundShipmentHeader - InboundShipmentHeader datatype
InboundShipmentItems - A list of InboundShipmentItem datatypes

the problem happens because InboundShipmentHeader is a datatype that takes another datatype as argument. In the end Amazon expects the following:

ShipmentId=102038383
InboundShipmentHeader.ShipmentName': 'somevalue',
InboundShipmentHeader.ShipFromAddress.Name': 'somevalue',
InboundShipmentHeader.ShipFromAddress.AddressLine1': 'somevalue',
InboundShipmentHeader.ShipFromAddress.City': 'somevalue',
InboundShipmentHeader.ShipFromAddress.StateOrProvinceCode': 'somevalue',
InboundShipmentHeader.ShipFromAddress.PostalCode': 'somevalue',
InboundShipmentHeader.ShipFromAddress.CountryCode': 'somevalue',
InboundShipmentHeader.DestinationFulfillmentCenterId': 'somevalue',
InboundShipmentHeader.ShipmentStatus': 'somevalue',
InboundShipmentHeader.LabelPrepPreference': 'somevalue',
InboundShipmentItems.member.1.QuantityShipped': 'somevalue',
InboundShipmentItems.member.2.QuantityShipped': 'somevalue',
InboundShipmentItems.member.1.SellerSKU': 'somevalue',
InboundShipmentItems.member.2.SellerSKU': 'somevalue',
InboundShipmentHeader.ShipFromAddress.AddressLine2': 'somevalue',
InboundShipmentHeader.ShipFromAddress.DistrictOrCounty': 'somevalue',

so I want to make it simple for someone to make this call without having to worry about the names of each argument. My solution is to create a base datatype class and then create the separate datatypes as classes.

This is what I have so far:

class AmazonDataType(dict):
    """
    Base for all Amazon datatypes.
    """

    def __init__(self, *args, **kwargs):
        self._prefix = kwargs.pop('prefix', '')
        self.update(*args, **kwargs)

    @property
    def prefix(self):
        return self._prefix

    @prefix.setter
    def prefix(self, value):
        self._prefix = value
        newdict = {'%s.%s' % (value, key): dictvalue for key, dictvalue in self.iteritems()}
        self.clear()
        dict.update(self, newdict)

    def __setitem__(self, key, value):
        try:
            original_key = self.fields[key]
        except KeyError, e:
            raise e
        if isinstance(value, AmazonDataType):
            value.prefix = original_key
            dict.update(self, value)
        else:
            newkey = self.prefix + original_key if self.prefix else original_key
            dict.__setitem__(self, newkey, value)

    def update(self, *args, **kwargs):
        """
        Props to Matt Anderson (http://stackoverflow.com/a/2390997/389453)
        """
        for k, v in dict(*args, **kwargs).iteritems():
            self[k] = v


class InboundShipmentHeader(AmazonDataType):
    fields = {
        'name': 'ShipmentName',
        'address': 'ShipFromAddress',
        'fulfillment_center_id': 'DestinationFulfillmentCenterId',
        'label_preference': 'LabelPrepPreference',
        'cases_required': 'AreCasesRequired',
        'shipment_status': 'ShipmentStatus',
    }

then instead of doing

somedict = {
    'InboundShipmentHeader.ShipmentName': 'somevalue',
    'InboundShipmentHeader.ShipFromAddress.Name': 'somevalue',
    'InboundShipmentHeader.ShipFromAddress.AddressLine1': 'somevalue',
    'InboundShipmentHeader.ShipFromAddress.City': 'somevalue',
    'InboundShipmentHeader.ShipFromAddress.StateOrProvinceCode': 'somevalue',
    'InboundShipmentHeader.ShipFromAddress.PostalCode': 'somevalue',
    'InboundShipmentHeader.ShipFromAddress.CountryCode': 'somevalue',
    'InboundShipmentHeader.DestinationFulfillmentCenterId': 'somevalue',
    'InboundShipmentHeader.ShipmentStatus': 'somevalue',
    'InboundShipmentHeader.LabelPrepPreference': 'somevalue',
}

call_amazon(somedict)

I want to pass something like

ShipmentHeader = InboundShipmentHeader()
ShipmentHeader['name'] = 'somevalue'
ShipmentHeader['address'] = address_datatype_instance
ShipmentHeader['fulfillment_center_id'] = 'somevalue'
ShipmentHeader['label_preference'] = 'somevalue'
ShipmentHeader['cases_required'] = 'somevalue'
ShipmentHeader['shipment_status'] = 'somevalue'

call_amazon(ShipmentHeader, otherparams)

In the background, the call_amazon method does:

ShipmentHeader.prefix = InboundShipmentHeader

Answer 1

You could subclass dict and add a method (I'm not sure what to call it, so let's say dict):

class AmazonDataType(dict):
    """
    Base for all Amazon datatypes.
    """

    def __init__(self, *args, **kwargs):
        self._prefix = kwargs.pop('prefix', self.__class__.__name__)

        super(AmazonDataType, self).__init__(*args, **kwargs)

    def __getattr__(self, key):
        return self.__getitem__(key)

    def __setattr__(self, key, value):
        return self.__setitem__(key, value)

    def dict(self):
        result = {}

        for key, value in self.items():
            if key.startswith('_'):
                continue

            key = self.fields.get(key, key)

            if isinstance(value, AmazonDataType):
                for skey, svalue in value.dict().items():
                    result['%s.%s' % (self._prefix, skey)] = svalue
            else:
                result['%s.%s' % (self._prefix, key)] = value

        return result

Now, the interface is a little more Pythonic:

class InboundShipmentHeader(AmazonDataType):
    fields = {
        'name': 'ShipmentName',
        'address': 'ShipFromAddress',
        'fulfillment_center_id': 'DestinationFulfillmentCenterId',
        'label_preference': 'LabelPrepPreference',
        'cases_required': 'AreCasesRequired',
        'shipment_status': 'ShipmentStatus',
    }

class Address(AmazonDataType):
    fields = {
        'name': 'Name',
        'address': 'AddressLine1',
        'city': 'City'
    }

address = Address(prefix='ShipFromAddress')
address.name = 'Foo'

header = InboundShipmentHeader()
header.name = 'somevalue'
header.address = address
header.fulfillment_center_id = 'somevalue'
header.label_preference = 'somevalue'
header.cases_required = 'somevalue'
header.shipment_status = 'somevalue'

The output of header.dict() is:

{'InboundShipmentHeader.AreCasesRequired': 'somevalue',
 'InboundShipmentHeader.DestinationFulfillmentCenterId': 'somevalue',
 'InboundShipmentHeader.LabelPrepPreference': 'somevalue',
 'InboundShipmentHeader.ShipFromAddress.Name': 'Foo',
 'InboundShipmentHeader.ShipmentName': 'somevalue',
 'InboundShipmentHeader.ShipmentStatus': 'somevalue'}