Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Retrieve/List objects using metadata in s3 - aws sdk

I have used User-Defined Metadata data to store the file in S3 bucket. Lets say my meta data would be like

metaData = {
    "title": "some random user title", 
    "description": "some random user description"
}

I understand that i can download file using the object key and the bucket name. I am looking whether there any way/options to get/retrieve/list the file by passing only the bucket name and User-Defined Metadata used for the object to upload in S3.

And also to know the actual usage of User-Defined Metadata

Any guidance or suggestions would greatly helpful.!

like image 291
Paul Avatar asked May 04 '18 12:05

Paul


People also ask

Can Athena query metadata S3?

This solution maintains an index in an Apache Parquet file, which optimizes Athena queries to search Amazon S3 metadata. Using this approach makes it straightforward to run queries as needed without the need to ingest data or manage any servers.

Which request returns the metadata information of an object?

The HeadObject action retrieves metadata from an object without returning the object itself. This operation is useful if you're only interested in an object's metadata. To use HEAD, you must have READ access to the object.

Can you get access logs for objects in Amazon Simple Storage Service Amazon S3?

By default, Amazon S3 doesn't collect server access logs. When you enable logging, Amazon S3 delivers access logs for a source bucket to a target bucket that you choose. The target bucket must be in the same AWS Region and AWS account as the source bucket, and must not have a default retention period configuration.


1 Answers

There is no native operation in Amazon S3 that can locate/retrieve objects via Metadata.

Metadata can be retrieved from objects, but cannot be used to search for objects. Also, only limited Metadata can be retrieved as part of an object listing.

An alternative would be to use Amazon S3 Inventory - Amazon Simple Storage Service. This only provides the listing once per day, but it can include metadata. You could use these files to locate objects based on metadata.

See:

  • Amazon S3 : Listing Obejct with Metadata in single request
  • Building and Maintaining an Amazon S3 Metadata Index without Servers | AWS Big Data Blog
like image 127
John Rotenstein Avatar answered Oct 17 '22 06:10

John Rotenstein