Best practices for storing references to AWS S3 objects in a database?

Question

We store files in Amazon AWS S3, and want to keep references to those files in a Document table in Postgres. I am looking for best practices. We use Python/Django, and currently store the URL that comes back from boto3.s3.key.Key().generate_url(...). But so many issues with that:

• Must parse the bucket and key out of the URL.
• Need to urldecode the key name.
• Doesn't support object versioning.
• Unicode support is easy to mess up, esp due to the urlencode/decode steps.

So, I'm considering storing the Bucket, Key, and Version in three separate fields, and creating the Key as a combination of the DB primary key plus a safely-encoded filename, but didn't know if there were better approaches?

Answer 1

Not sure if best-est approach, but we store:

• unique object ID (might be UUID) in database (for which Postgres has a native UUID type)
• bucket name and path in configuration (as we store all the objects of the same type under the same bucket+path)

That way you can at least:

• Move objects to a different bucket / path without havig to rewrite your whole database table
• Switch from S3 to local storage if you choose so
• Throw away your primary key (e.g. while partitioning tables) without loosing track of your objects