Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

AWS S3: how to list object by tags [duplicate]

We have our bucket with new Aws SDK API on AWS S3. We uploaded and tagged lots of files and folders with tags.

How can we filter on key-value tag, or only one of them? I'd like to find all the objects with key = "temp", or key = "temp" and value = "lol".

Thanks!

like image 335
alagaesia Avatar asked Jan 10 '17 14:01

alagaesia


People also ask

Can S3 objects have tags?

You can add tags to an Amazon S3 object during the upload or after the upload. Each tag is a key-value pair that adheres to the following rules: You can associate up to 10 tags with an object they must have unique tag keys.

How does S3 handle duplicate file names?

S3 Storage does not allow duplicate file names, but the AttachContent function was allowing duplicate file attachments. Due to this, the sync was failing to attach on the server and the message was never communicated to the client.

Can S3 buckets have tags?

Each S3 bucket has a tag set. A tag set contains all of the tags that are assigned to that bucket. A tag set can contain as many as 50 tags, or it can be empty. Keys must be unique within a tag set, but values in a tag set don't have to be unique.

How many tags can an S3 object have assigned to it?

You can associate up to 10 tags with an object. Tags associated with an object must have unique tag keys.


2 Answers

I also hoped that AWS will eventually support "search files by tags" because that would open up possibilities like e.g. having a photo storage with the names, descriptions, location stored in tags so I wouldn't need a separate database.

But, apparently AWS explicitly is not supporting this, and will probably never do so. Quoting from their storage service white paper:

Amazon S3 doesn’t suit all storage situations. [...] some storage needs for which you should consider other AWS storage options [...]

Amazon S3 doesn’t offer query capabilities to retrieve specific objects. When you use Amazon S3 you need to know the exact bucket name and key for the files you want to retrieve from the service. Amazon S3 can’t be used as a database or search engine by itself.

Instead, you can pair Amazon S3 with Amazon DynamoDB, Amazon CloudSearch, or Amazon Relational Database Service (Amazon RDS) to index and query metadata about Amazon S3 buckets and objects.

AWS suggests using DynamoDB, RDS or CloudSearch instead.

like image 182
hansaplast Avatar answered Sep 21 '22 14:09

hansaplast


Seems like this is now possible using AWS Resource Tagging API and S3 Select (SQL). See this post: https://aws.amazon.com/blogs/architecture/how-to-efficiently-extract-and-query-tagged-resources-using-the-aws-resource-tagging-api-and-s3-select-sql/

like image 31
Malgi Avatar answered Sep 20 '22 14:09

Malgi