S3 data partitioning for bucket logging files

Question

I have a s3 bucket "ABC" and logging is enabled at this bucket and logs stored in "ABC-logs". There are many files comes in "ABC-logs" per day. Now I want to segregate these logs year wise. For ex:

s3://ABC-logs/year=2015
s3://ABC-logs/year=2016
s3://ABC-logs/year=2017

What is the best way to do this. I thought to do it via awscli but each year end, I will have to change bucket logging folder.

Answer 1

The traditional way to do this is via an Amazon EMR cluster.

You can use Hive to create an EXTERNAL TABLE pointing at your source data. Then, create another EXTERNAL TABLE pointing to where you would like to store the data, with appropriate parameters such as Partition and filetype. Then, select data from the first table and insert it into the destination table.

Partitioned data is more efficient for querying since whole folders can be skipped if it is known that the data is not necessary for the query.

Answer 2

S3 supports events, so create and set the logs to the folder say newObjects s3://ABC-logs/newObjects and enable a event with newobjects prefix (/newObjects) with lambda notification , Inside lambda receive the object key and move it with the current year prefix like (s3://ABC-logs/year=2017/objectname).