Get last N records in a DynamoDB table

Question

Is there any way to get the last N records from a dynamodb table. The range key I have is the timestamp. So I could use the ScanIndex forward to order items chronologically.

But in order to query I need to have a hashKey condition, which I don't want to filter. Any thoughts?

Answer 1

DynamoDB is not designed to work this way. The items are distributed according to a hash on the HashKey in such a way that the order is not predictable.

Your options include:

• grouping the items under a single hash key (not recommended: you would overload a few servers with your data, and Amazon cannot guarantee your read/write capacity)
• scanning the whole table and keep the N most recent items (something like for (item in items) { if (item newer then oldest accumulated item) accumulate item; });
• partition your table into multiple tables (ie, instead of a table called Events, create one called Events20130705 for today's events, Events20130706 for tomorrow's events), and scan just like the previous option -- this way your scans are smaller

You could also maybe change your data model. For example, You could have one versioned entry that would keep references to the N most recent items. Or you could have something like a single counter that you'd increment and update N other entries under hashkeys such as recent-K where K is your counter mod N.

Maybe you could even use another tool for this job. For instance, you could have a Redis server to do this. Without knowing your use case with much more detail, it is hard to make a precise suggestion -- how scalable should this be? how reliable should it be? how much maintenance are you willing to perform? how much are you willing to pay for it?

It's usually better to embrace the limitation, know your constraints and be creative.