Python Boto DynamoDB: Retrieve Only Specified Attributes From A Record

Question

I'm trying to tune performance for a high volume site. I'm using the boto.dynamodb2 libraries. How can I retrieve only the specified attributes from a record given it's primary hash key (no range)?

As a secondary question, will this affect how we are billed by amazon? Amazon charged per record or per Kilobyte of data read (whichever unit is larger). If, for a given record, I have 3 attributes that only total 100 bytes, and a 4th a attribute that totals 63KB, I request only the first 3 (smaller) attributes. Do I get charged for 1 unit, or am I still charged for 64 units?

Answer 1

From boto's documentation for get_item you can pass attributes_to_get list:

attributes_to_get (list) – A list of attribute names. If supplied, only the specified attribute names will be returned. Otherwise, all attributes will be returned.

Re secondary question sadly you can't "save" reads by skipping attributes. From the DynamoDB's documentation:

For any operation that returns items, you can request a subset of attributes to retrieve; however, doing so has no impact on the item size calculations.

Also note that item size is limited to 64KB (so you can't have an attribute of 300KB)

Answer 2

You can look into Local Secondary Index on DynamoDB. For Local Secondary Index you can specify the same hash key as your primary key, and a specific list of attributes to retrieve, you will be billed for those attributes only. The size of the index will count toward DynamoDB storage allocation, which is not that expensive, IMHO.

When you choose the attributes to project into a local secondary index, you must consider the tradeoff between provisioned throughput costs and storage costs:

• If you need to access just a few attributes with the lowest possible latency, consider projecting only those attributes into a local secondary index. The smaller the index, the less that it will cost to store it, and the less your write costs will be. If there are attributes that you occasionally need to fetch, the cost for provisioned throughput may well outweigh the longer-term cost of storing those attributes.

• If your application will frequently access some non-key attributes, you should consider projecting those attributes into a local secondary index. The additional storage costs for the local secondary index will offset the cost of performing frequent table scans.

• If you need to access most of the non-key attributes on a frequent basis, you can project these attributes—or even the entire source table— into a local secondary index. This will give you maximum flexibility and lowest provisioned throughput consumption, because no fetching would be required; however, your storage cost would increase, or even double if you are projecting all attributes.

• If your application needs to query a table infrequently, but must perform many writes or updates against the data in the table, consider projecting KEYS_ONLY. The local secondary index would be of minimal size, but would still be available when needed for query activity.

From An Introduction to boto’s DynamoDB v2 interface:

You can also run queries against the local secondary indexes. Simply provide the index name (index='FirstNameIndex') & filter parameters against its fields:

# Users within the last hour.
>>> recent = users.query(
...     account_type__eq='standard_user',
...     date_joined__gte=time.time() - (60 * 60),
...     index='DateJoinedIndex'
... )

>>> for user in recent:
...     print user['first_name']
'Alice'
'Jane'