How do I expire keys in dynamoDB with Boto?

Question

I'm trying to move from redis to dynamoDB and sofar everything is working great! The only thing I have yet to figure out is key expiration. Currently, I have my data setup with one primary key and no range key as so:

{
  "key" => string,
  "value" => ["string", "string"],
  "timestamp" => seconds since epoch
}

What I was thinking was to do a scan over the database for where timestamp is less than a particular value, and then explicitly delete them. This, however, seems extremely inefficient and would use up a ridiculous number of read/write units for no reason! On top of which, the expirations would only happen when I run the scan, so they could conceivably build up.

So, has anyone found a good solution to this problem?

Answer 1

I'm also using DynamoDB like the way we used to use Redis.

My suggestion is to write the key into different time-sliced tables.

For example, say a type of record should last few minutes, at most less an hour, then you can

1. Create a new table every day for this type of record and store new records in today's table.
2. Use a read repair tip when you read records, which means if you can't find a record in today's table, you try to find it in yesterday's table and put in today's table if necessary.
3. If you find the record in either table, verify it with it's timestamp. It's not necessary to delete expired records at this moment.
4. Drop entire stale tables in your tasks.

This is easier to maintain and cost-efficient.

Answer 2

You could do lazy expiration and delete it on request.

For example:

• store key "a" with an attribute "expiration", expires in 10 minutes.
• fetch in 9 minutes, check expiration, return it.
• fetch in 11 minutes. check expiration. since it's less than now, delete the entry.

This is what memcached was doing when I looked at the source a few years ago.

You'd still need to do a scan to remove all the old entries.

You could also consider using Elasticache, which is meant for caching rather than a permanent data store.

Answer 3

It seems that Amazon just added expiration support to DynamoDB (as of feb 27 2017). Take a look at the official blog post:

https://aws.amazon.com/blogs/aws/new-manage-dynamodb-items-using-time-to-live-ttl/

Answer 4

You could use the timestamp as the range key which would be indexed and allow for easier operations based on the time.