Can make figure out file dependencies in AWS S3 buckets?

Question

The source directory contains numerous large image and video files.

These files need to be uploaded to an AWS S3 bucket with the aws s3 cp command. For example, as part of this build process, I copy my image file my_image.jpg to the S3 bucket like this: aws s3 cp my_image.jpg s3://mybucket.mydomain.com/

I have no problem doing this copy to AWS manually. And I can script it too. But I want to use the makefile to upload my image file my_image.jpg iff the same-named file in my S3 bucket is older than the one in my source directory.

Generally make is very good at this kind of dependency checking based on file dates. However, is there a way I can tell make to get the file dates from files in S3 buckets and use that to determine if dependencies need to be rebuilt or not?

Answer 1

The AWS CLI has an s3 sync command that can take care of a fair amount of this for you. From the documentation:

A s3 object will require copying if:

• the sizes of the two s3 objects differ,
• the last modified time of the source is newer than the last modified time of the destination,
• or the s3 object does not exist under the specified bucket and prefix destination.